Bidirectional fusion image style migration method and system based on content and style decoupling

Abstract

The invention relates to the technical field of image conversion, and provides a bidirectional fusion image style migration method and system based on content and style decoupling, and the method comprises the steps: obtaining a style image set and a content image, respectively extracting a common style feature and a content feature, carrying out the bidirectional fusion of the style feature and the content feature through a cross attention mechanism, and carrying out the bidirectional fusion of the style feature and the content feature. Generating two-way fusion features after fusion; extracting a text feature based on the obtained text description, splicing the text feature with the bidirectional fusion feature to generate a condition control vector as guide information for image generation, executing a denoising inverse diffusion process guided by the condition control vector on the obtained target image, and finally decoding to obtain a stylized image; based on bidirectional feature fusion of styles and contents and a text-guided diffusion generation mechanism, an image style migration effect with delicate style expression, clear semantic structure and strong controllability is realized.

Description

Technical Field

[0001] The present invention relates to the technical field related to image conversion, and in particular to a bidirectional fusion image style transfer method and system based on content and style decoupling. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of visual creative industries such as image editing, digital content creation, and fashion design, style transfer technology has been widely used to achieve the fusion and reconstruction of image style and content. In fashion design scenarios in particular, it is necessary to effectively transfer the visual style of the style image, such as brushstrokes, texture, and color, to the target image while maintaining the content structure and semantic consistency of the target image, thus achieving the unity of style expression and content preservation. However, the style transfer task inherently involves cross-domain feature alignment and fusion, requiring both the capture of rich, fine-grained style information and the precise preservation of the target image's content structure. This places higher demands on the model's feature modeling capabilities and fusion mechanisms.

[0004] Although current style transfer methods have achieved certain results in various application scenarios, they still face some problems when dealing with highly complex and demanding image style transfer tasks: imbalanced fusion of content and style, feature entanglement, and insufficient style expression: First, most methods use a one-way fusion strategy for feature fusion, using content features to dominate style injection while ignoring the reverse effect of style features on content structure. This leads to an imbalanced fusion effect, a disconnect between content and style, and difficulty in achieving a coordinated and unified visual effect. For example, methods based on generative adversarial networks (GANs) can achieve image conversion without the need for paired training data, but they suffer from problems such as insufficient expression of style details, easy distortion of semantic structure, and an imbalance in the fusion of content and style. Secondly, both Generative Adversarial Networks (GANs) and diffusion models often experience "feature entanglement," where content and style features interfere with each other, during the transfer process. This occurs when stylistic traces from the content image or semantic structure from the style image are mixed into the output, affecting the image's semantic clarity and style purity. For example, methods based on large-scale pre-trained diffusion models (such as the Stable Diffusion Model (SDM)) leverage a large number of parameters and extensive training data to generate highly realistic images while maintaining semantic consistency. Related research has further introduced textual cues as conditional control information to improve the artistic quality and controllability of generated images. Rich text editors have also been proposed, which, by extracting more detailed textual cues to constrain content structure, mitigate the problem of content structure drift to some extent. However, these methods may still introduce style image content unrelated to the target content during the style transfer process, resulting in a disruption of the content structure.

[0005] Finally, due to their limited ability to extract and express fine-grained style information (such as brushstrokes, textures, and color schemes), existing methods often fail to fully express the style of complex image synthesis, making it difficult to achieve refined and consistent transfer results. These issues are particularly prominent in applications such as fashion design, which require high style accuracy, and thus limit the practicality and dissemination value of style transfer technology. Summary of the Invention

[0006] In order to solve the above problems, the present invention proposes a bidirectional fusion image style transfer method and system based on content and style decoupling. Based on the bidirectional feature fusion of style and content and the text-guided diffusion generation mechanism, it achieves an image style transfer effect with exquisite style expression, clear semantic structure and strong controllability.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions: One or more embodiments provide a bidirectional fusion image style transfer method based on content and style decoupling, comprising the following steps: Obtain a set of style images and content images, extract common style features and content features respectively, and use a cross-attention mechanism to bidirectionally fuse the style features and content features to generate a fused bidirectional fusion feature; Text features are extracted based on the acquired text description and spliced ​​with the bidirectional fusion features to generate a conditional control vector as the guiding information for image generation. A denoising inverse diffusion process guided by the conditional control vector is performed on the acquired target image, and finally a stylized image is decoded.

[0008] One or more embodiments provide a bidirectional fusion image style transfer system based on content and style decoupling, including: The bidirectional fusion module for common style enhancement is configured to obtain a set of style images and content images, extract common style features and content features respectively, and bidirectionally fuse the style features and content features using a cross-attention mechanism to generate a fused bidirectional fusion feature. The conditionally guided diffusion module is configured to extract text features based on the acquired text description, and concatenate them with the bidirectional fusion features to generate a conditional control vector as guidance information for image generation. The denoising inverse diffusion process guided by the conditional control vector is performed on the acquired target image, and finally a stylized image is decoded.

[0009] One or more embodiments provide a bidirectional fusion image style transfer system based on content and style decoupling, including: Image acquisition device and processor; The processor is configured to execute the steps of the above-mentioned bidirectional fusion image style transfer method based on content and style decoupling.

[0010] Compared with the prior art, the present invention has the following beneficial effects: The present invention realizes bidirectional feature fusion through the cross-attention mechanism, which improves the problem of inconsistent fusion of content and style in traditional methods, so that the output image presents a more detailed style expression while retaining the semantic structure; the introduction of the style image set improves the robustness of style feature extraction, avoiding the instability of style expression caused by differences in individual samples; the conditional control vector is constructed by using text features and bidirectional fused features to perform fine control of the image generation process at the semantic level, effectively suppressing the "feature entanglement" problem and improving the accuracy and controllability of style transfer; combined with the gradual denoising characteristics of the diffusion model, the generated image has higher authenticity and detail integrity; the overall solution can be widely applied to image generation scenarios with fine style requirements and clear semantics.

[0011] The advantages of the present invention and its additional aspects will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which constitute a part of the specification of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their description are used to explain the present invention but do not constitute a limitation of the present invention.

[0013] Figure 1 1 is a schematic diagram of the overall processing flow of bidirectional fusion image style transfer based on content and style decoupling according to Example 1 of the present invention; Figure 2 1 is a schematic structural diagram of a bidirectional fusion module for common style enhancement according to Example 1 of the present invention; Figure 3 Schematic diagram of the bidirectional fusion process of common style enhancement in Example 1 of the present invention; Figure 4 is a comparison chart of ablation experiment results of Example 1 of the present invention; Figure 5 2 is a comparison diagram of the influence of the weight parameter ζ on the balance between content and style in Example 1 of the present invention; Figure 6 Flowchart of a bidirectional fusion image style transfer method based on content and style decoupling according to embodiment 1 of the present invention; DETAILED DESCRIPTION The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0014] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0015] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments of the present invention and the features in the embodiments can be combined with each other. The embodiments will be described in detail below with reference to the accompanying drawings.

[0016] Example 1 In the technical solutions disclosed in one or more embodiments, Figures 1 to 6 As shown in FIG, the bidirectional fusion image style transfer method based on content and style decoupling includes the following steps: Step 1: Obtain a set of style images and content images, extract common style features and content features respectively, and use a cross-attention mechanism to bidirectionally fuse the style features and content features to generate a fused bidirectional fusion feature. Step 2: Extract text features based on the acquired text description and combine them with the bidirectional fusion features to generate a conditional control vector as guidance information for image generation. Perform a denoising inverse diffusion process guided by the conditional control vector on the acquired target image, and finally decode it to obtain a stylized image. This implementation achieves bidirectional feature fusion through a cross-attention mechanism, significantly improving the inconsistency between content and style fusion in traditional methods. This allows the output image to present a more detailed style expression while preserving the semantic structure. The introduction of a set of style images improves the robustness of style feature extraction, avoiding instability in style expression caused by individual sample differences. Conditional control vectors are constructed using text features to finely control the image generation process at a semantic level, effectively suppressing the "feature entanglement" problem and improving the accuracy and controllability of style transfer. Combined with the gradual denoising properties of the diffusion model, the generated image has higher realism and detail integrity. The overall solution is widely applicable to image generation scenarios requiring detailed style and clear semantics. In step 1, this is achieved by constructing a bidirectional fusion module with common style enhancement. First, bidirectional fusion is processed through a bidirectional attention mechanism of "style to content" and "content to style", effectively balancing content preservation and style transfer. Current methods often have difficulty capturing fine-grained details during the transfer of complex style images. Images of the same artistic style (such as Van Gogh's style) have inherent common features; fusing these common features can produce richer and more representative style representations. Based on this, this embodiment designs a bidirectional fusion module with common style enhancement to generate fused composite features; The goal of this embodiment is to transfer the style of the style image to the content image to achieve the regeneration of the content image; then the visual features of the content image (such as color, brushstrokes and texture) and the style features of the style image can be transferred to the target image, which can be an image of a fashion item; for example, Figure 1 The example in the figure converts and generates T-shirt images (fashion item images); In step 1, the style image set and content image are obtained, and the common style feature representation and content feature are extracted respectively. The cross attention mechanism is used to perform a bidirectional fusion of the style feature representation and content feature to generate the fused bidirectional fusion feature, such as Figure 2 and Figure 3 As shown in FIG, the method for generating bidirectional fusion features includes the following steps: Step 11: Extract features from the acquired style image set to obtain common style features. , enhanced by self-attention, and the enhanced style common features are obtained ; Step 111: Extract common style features from the acquired style image set , and perform weighted average to obtain the common features of the style the expression; Images of the same style have inherent commonalities, such as Van Gogh style. Fusion of these images can produce richer and more representative style features. From a set of images with the same style The style features are extracted and then fused by weighted averaging to derive the common style features. Expression, style common features It is a richer and more representative style feature; Step 112: Figure 3 As shown, the self-attention mechanism is used to integrate the common features of the style The representation of is encoded as the style query vector , style key vector and style value vector , get the attention map ; (1); in, It is a learnable parameter matrix; it can be encoded after being processed by normalization (Norm) and convolution operation (1×1Conv); Attention Map It can be calculated as: (2); Wherein, the superscript T indicates transposition; Step 113: According to the attention map The attention weights extracted from the style value vector The features in the weighted form are weighted to obtain the attention weighted average of the style features. ; Attention-weighted average is a representation that highlights the most relevant style features, as shown below: (3); Step 114: Based on the calculated attention map and style value vector and attention-weighted average , calculate the attention-weighted standard deviation ; Specifically, in order to further enhance the representation of style features and reduce local distortion, the attention weighted standard deviation is derived in this embodiment. , to better capture the statistical local characteristics of style features, the method is as follows: (4); Step 115: Attention-weighted standard deviation and average , the common characteristics of style Enhance and obtain the common features of the enhanced style , as shown below: (5); Step 12: Enhance the common features of the style Self-attention encoding is used to enhance the style query vector , style key vector and style value vector ; Step 13: Extract features from the acquired content image and use the attention mechanism to encode it into a content query vector. , content key vector and content value vector ; Specifically, it can be derived in the same way as described in formula (1) of 、 and , and the content features of the content image The query vector , key vector Sum vector .

[0017] Step 14: Based on the obtained attention mechanism vector, a cross attention mechanism is used to perform bidirectional fusion from content to style and from style to content to obtain a bidirectional fusion feature. , including the following process: Step 141: Fusion from content to style: transform the content query vector , enhanced style key vector and style value vector Perform cross attention operation (CA) to obtain the first fusion feature ; Fusion from content to style enables content features to guide style features. This process allows precise adjustment of style features according to content requirements and dynamically optimizes the representation of style features, thereby effectively enhancing the model's adaptability to different content feature requirements. The fusion formula is as follows: (6); Step 142: Fusion from style to content: The enhanced style query vector , content key vector and content value vector Perform cross attention operation (CA) to obtain the second fusion feature ; One-way fusion lacks the step of style to content. Therefore, in order to balance content and style features, this step uses style query based on the feature fusion guided by content features. and the key for the content feature Sum Perform attention operation (CA). The focus of this stage is to adjust the content features under the guidance of style features. The formula is as follows: (7); Step 143: Perform weighted fusion on the first fusion feature and the second fusion feature to obtain a bidirectional fusion feature. , as shown below: (8); in, It's balance and Contribution weight parameter.

[0018] This process can be expressed as: ; in, Indicates two-way fusion; In the above implementation, to address the difficulties existing methods face in capturing fine-grained style information in complex style image transfer, as well as the disharmony between content and style caused by a single feature fusion strategy, a bidirectional fusion mechanism for enhancing common style is proposed. First, by extracting and fusing style features from a set of similar style images, a more representative and robust common style representation is constructed. Subsequently, a weighted average and weighted standard deviation calculation method based on an attention mechanism is introduced to enhance the local expressive power of style features, improving the sophistication and consistency of style transfer. During the feature fusion stage, a bidirectional cross-attention strategy, with content guiding style and style guiding content, is employed to achieve semantic coordination and complementarity between content and style features, thereby addressing the weak style dominance and distorted content structure inherent in traditional one-way fusion strategies. This bidirectional fusion mechanism enables the model to achieve precise fusion of style and content in both structure and visual representation, ensuring complete and consistent style expression. The generated images are visually more coordinated, unified, and aesthetically consistent, offering significant advantages in complex style transfer scenarios.

[0019] In step 2, the condition-guided inverse diffusion process: extracting text features based on the acquired text description, splicing them with the bidirectional fusion features to generate a conditional control vector as guidance information for image generation, performing a denoising inverse diffusion process guided by the conditional control vector on the acquired target image, and finally decoding to obtain a stylized image. The method includes the following steps: Step 21: Use the text encoder to extract text features from the obtained text description to obtain the text embedding vector ; like Figure 1The natural language description shown is, for example, "a Van Gogh-style fashion item with a blue sky and white cloud pattern." The natural language description provides information about both the content "blue sky and white cloud pattern" and the style "Van Gogh style."

[0020] Optionally, use CLIP Text Encoder to encode the input text to obtain a text embedding vector ; Step 22: Embed the text vector with the bidirectional fusion feature After concatenation, it is fed into the Text Transformer model to generate the conditional control vector C; Step 23: Perform the denoising inverse diffusion process guided by the conditional control vector on the acquired target image to obtain the final latent variable , including the following steps: Step 231: Use the ReNoise strategy to gradually add noise to the target image, then predict the noise through inverse diffusion denoising, and add the predicted noise to the target image to obtain a noisy image. ; The process of gradually adding noise is realized by the noise adding module, which is the forward process in the diffusion process; The target image is a fashion item image. In this embodiment, a T-shirt image is used as an example. The content image includes a blue sky and white clouds image, which is converted into the style of Van Gogh to obtain the final migrated target image. Different from the traditional diffusion model, the starting point of the inversion using the diffusion method in this embodiment is the predicted noise rather than the random noise.

[0021] Step 232: Condition-guided denoising diffusion process: Input into the Denoising U-Net module, and perform step-by-step denoising based on the conditional vector guidance to obtain the final latent variable ; This step is the reverse process of the diffusion process; Denoising U-Net is a neural network used in the reverse phase of the diffusion process (denoising process). Its main function is to restore the original image from the noisy image, or at least predict the noise added to the image. It includes processes such as predicting noise and adding predicted noise for conditional diffusion. The Denoising U-Net module is based on the U-Net architecture and includes: Encoder: extracts multi-scale features of the image and gradually downsamples it; Decoder: restores high-level semantic information to image resolution and gradually upsamples; Skip Connections: connect encoders and decoders at the same level to preserve detailed information; Time Embedding: This module is used to introduce information about the number of diffusion steps t, which is used to determine the current stage of the image denoising process. Conditional Embedding: used to introduce conditional information, such as categories, text descriptions, etc. Step 24, image decoding: the final latent variable , decoded by the decoder to obtain a stylized image; Furthermore, the above-mentioned noise addition module and the Denoising U-Net module constitute an SDM diffusion model; in the pre-trained SDM diffusion model, in order to better retain the content of the fashion item during the back propagation process, this embodiment adopts a new inversion method called ReNoise. Specifically, the method first gradually adds noise to the fashion item image during the forward noise processing process to obtain a noisy image. Subsequently, denoising is performed using a diffusion model, and the noise in the image is predicted during the reverse denoising process. The predicted noise as the initial potential signal is input into the SDM for further denoising. At the same time, in order to further enhance content and style guidance, this embodiment uses the CLIP text encoder to generate text embeddings , and then compare it with Combined with text converter to generate conditional control information In tensor Under the regulation of , the diffusion process is directed to the desired style while preserving the original content.

[0022] Furthermore, the optimization objective of the SDM diffusion model during training is: (9); in, ; It is fixed during training, representing the operation of a neural network with frozen parameters; Represents the content image, Represents a style image; Target image after adding noise; Conditional control information Under the guidance of , the trained SDM diffusion model generates a stylized image, that is, the target image after style transfer is obtained. .

[0023] In the above implementation, by introducing the ReNoise inversion mechanism, noise is gradually added to the input fashion item image in the forward process to obtain a noisy image under Gaussian perturbation; then, in the reverse denoising process, not only is the noise distribution predicted to restore the original content, but the predicted initial potential signal is also used as a key input to guide the subsequent generation process, so that the model can better maintain the semantic structure and visual features of the original image while reconstructing the style. Further, a conditional control tensor is generated through text data to adjust the generation path of the entire diffusion process so that it can accurately point to the target style expression direction while maintaining content consistency. This method not only improves the model's responsiveness to the target style, but also enhances the consistency, interpretability and interactivity of image generation. Through the collaborative design of ReNoise inversion and text conditional control mechanisms, the proposed technical solution can achieve more accurate content retention, more natural style transfer, and stronger generation controllability; A further technical solution is that the process from step 1 to step 2 is achieved by constructing a style transfer network, including a bidirectional fusion module for common style enhancement and an SDM diffusion model. The bidirectional fusion module for common style enhancement includes a content extractor, a style extractor, and an SBF module. The content extractor is used to extract content features from the content image; the style extractor is used to extract style features; the SBF module is used to perform bidirectional fusion of style features and content features; The SDM diffusion model is configured to extract text features based on the acquired text description, concatenate them with the bidirectional fusion features to generate a conditional control vector as guidance information for image generation, and then perform a denoising inverse diffusion process guided by the conditional control vector on the acquired target image to ultimately decode the stylized image. Further technical solutions also include using a contrast optimization strategy based on the decoupling of content and style features to train the style transfer network, including the following steps: Step S1: Obtain the style image, content image, and target image after style transfer , build a training image set; Style images, such as different Van Gogh-style works; content images, such as images of target fashion items; Step S2: Feature extraction and decoupling: using style extractor Extract the target image after style transfer Style characteristics , and the style characteristics of the content image ; Using content extractor Extract the target image after style transfer Content features in , and Content characteristics of style images ; Target image after style transfer Content Features and style characteristics The extraction formula is: (10); Similarly, you can extract and , respectively represent the style features of the content image and the Content characteristics of style images: (11); Step S3: Use style extractor Extract the style features of style images of the same style and perform weighted average to obtain the common style features ; Using content extractor Extract content features of content images ; Step S4: Construct style contrast learning and transfer the style to the target image Style characteristics Common features with style Construct a positive sample pair; the target image after style transfer Style characteristics Style features of content images Constructed as a negative sample pair; In this embodiment, the is a positive sample pair, is a negative sample pair. This stage can improve the model's ability to identify the style features of the reference image. The style consistency loss during training is: (12); in, is the first temperature coefficient. In a specific feature space, contrastive learning can make the target style feature and the style common features of the style image The distance between them decreases.

[0024] Step S5: Construct content contrast learning to transfer the target image after style transfer Content Features and content characteristics Construct a positive sample pair; the target image after style transfer Content Features Content features of style images Constructed as a negative sample pair; In order to minimize the content characteristics of the target content and content image A content contrast loss is designed. Considered as a positive sample pair, The goal is to improve the model's ability to identify the content features of the reference image, and the constructed content fidelity loss is: (13); in, is the second temperature coefficient.

[0025] Step S5, performing comparative learning training based on the constructed positive sample pairs and negative sample pairs, optimizing the parameters of the style transfer network based on the constructed overall loss function, and obtaining a trained style transfer network; Furthermore, the overall loss includes condition control losses , style consistency loss and content fidelity loss , the total loss function is expressed as follows: (14); in, is a hyperparameter that balances the contrastive loss.

[0026] In traditional style transfer generation tasks, there is often representational entanglement between the style features of the content image and the content features of the style image, making it difficult for the model to effectively distinguish and extract pure content and style information, which in turn affects the style consistency and structural fidelity of the generated image.

[0027] The contrastive optimization strategy based on decoupling content and style features proposed in the above implementation introduces a dual contrastive learning mechanism during training, explicitly constraining both content and style representations, guiding the model to enhance its ability to separate and identify these two types of features. This strategy significantly improves the model's ability to decouple content and style modeling, enabling the generated results to more accurately preserve the original content structure while simultaneously achieving precise transfer of style features.

[0028] In order to verify the effectiveness and effect of the above method of this embodiment, a simulation experiment was carried out; Conduct ablation experiments to construct a variant of the style transfer network of this embodiment: w / o-SBF variant: replace the SBF module with AdaAttN to verify the effectiveness of the SBF module; Figure 4 (e) is the effect of the image after style transfer using the w / o-SBF variant; w / o-Commonality variant: In order to verify the impact of the commonality style enhancement bidirectional fusion module on the model of this embodiment, a single style image instead of multiple style images is used for fashion style transfer;Figure 4 (d) shows the effect of the image after style transfer using the w / o-Commonality variant; w / o-Contrastive variant: In order to evaluate the importance of the contrastive optimization strategy in the model of this embodiment, the training method of the contrastive optimization strategy was deleted. Figure 4 (f) is the effect of the image after style migration using the w / o-Commonality variant; Figure 4 (g) is the effect diagram of the image processed by the method of this embodiment. The overall model of this embodiment is used, that is, Figure 1 A complete style transfer network model is built, and a contrast optimization strategy based on the decoupling of content and style features is used to train the style transfer network model, and the effect of image style transfer is obtained; Figure 4 (a) is a Van Gogh style image. Figure 4 (b) in the figure is the target image. The clothing image is used as the target image for design. Figure 4 (c) in the figure is the content image; The method of this embodiment outperforms all variants, demonstrating the effectiveness of each component in the method of this embodiment. Specifically: (1) Compared with the results generated by the original style transfer network of this embodiment, the stylized results of the w / o-SBF variant show more obvious artistic style, but the content preservation is relatively poor. This highlights the key role of the SBF module in achieving a harmonious balance between content and style. (2) The w / o-Commonality variant uses a single style image for conversion, but the generated image does not achieve the expected style effect. This shows that the scheme of this embodiment aggregates the style features of multiple images in the same style category, which can enhance the accurate capture and expression of style commonality. (3) The results generated by the w / o-Contrastive variant introduce patterns that are unrelated to the content image and a style that deviates from the target style. These findings further confirm that the contrast optimization strategy based on the decoupling of content and style features can enhance the model's ability to distinguish between content and style.

[0029] In addition to the field of fashion style transfer, the method of this embodiment can also be applied to the field of artistic style transfer. In order to evaluate the performance of the style transfer network of this embodiment, the DIV2K dataset is used for the style transfer task.

[0030] exist Figure 5 The generated images are analyzed with the parameters Changes in parameters It's balance and The key factor of contribution, Content represents the content image. The increase in size amplifies the artistic style but also leads to a loss of image content, and vice versa. Figure 5 (a), (b), (c), and (d) are the processing results of four content images (Content); for example, Figure 5 (d) in the figure shows that when = 0.2, the squirrel’s appearance is very similar to the content image. In contrast, when = 1.0, the artistic style is very obvious, but the image content is significantly reduced. It is worth noting that despite the strong artistic style of the latter, the outline of the squirrel is still preserved. In addition, it is observed that when =0.6, the overall performance is the best.

[0031] Example 2 Based on Example 1, this embodiment provides a bidirectional fusion image style transfer system based on content and style decoupling, including: The bidirectional fusion module for common style enhancement is configured to obtain a set of style images and content images, extract common style features and content features respectively, and bidirectionally fuse the style features and content features using a cross-attention mechanism to generate a fused bidirectional fusion feature. The conditionally guided diffusion module is configured to extract text features based on the acquired text description, and concatenate them with the bidirectional fusion features to generate a conditional control vector as guidance information for image generation. The denoising inverse diffusion process guided by the conditional control vector is performed on the acquired target image, and finally a stylized image is decoded.

[0032] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation processes are the same, which will not be repeated here.

[0033] Example 3 Based on Example 1, this embodiment provides a bidirectional fusion image style transfer system based on content and style decoupling, including: Image acquisition device and processor; The image acquisition device is used to acquire style images and content images; The processor is configured to execute the steps of the bidirectional fusion image style transfer method based on content and style decoupling described in Example 1.

[0034] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

[0035] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A bidirectional fusion image style transfer method based on content and style decoupling, characterized by: The steps include: Obtain a set of style images and content images, extract common style features and content features respectively, and use a cross-attention mechanism to bidirectionally fuse the style features and content features to generate a fused bidirectional fusion feature; Text features are extracted based on the acquired text description and spliced ​​with the bidirectional fusion features to generate a conditional control vector as the guiding information for image generation. A denoising inverse diffusion process guided by the conditional control vector is performed on the acquired target image, and finally a stylized image is decoded.

2. The method for bidirectional fusion image style transfer based on content and style decoupling according to claim 1, characterized in that: The method for generating bidirectional fusion features includes the following steps: For the acquired style image set, feature extraction is performed to obtain the common style features , enhanced by self-attention, and the enhanced style common features are obtained ; Will enhance the common characteristics of style Self-attention encoding is used to enhance the style query vector, style key vector and style value vector; Extract features from the acquired content image and use the attention mechanism to encode it into content query vector, content key vector, and content value vector; Based on the obtained attention mechanism vector, a cross attention mechanism is used to perform bidirectional fusion from content to style and from style to content to obtain a bidirectional fusion feature .

3. The method for bidirectional fusion image style transfer based on content and style decoupling according to claim 2, characterized in that: Based on the obtained attention mechanism vector, a cross attention mechanism is used to perform bidirectional fusion from content to style and from style to content to obtain a bidirectional fusion feature , including the following process: Content query vector , enhanced style key vector and style value vector Perform cross-attention operation to obtain the first fusion feature; The enhanced style query vector , content key vector and content value vector Perform cross-attention operation to obtain the second fusion feature; The first fusion feature and the second fusion feature are weighted and fused to obtain a bidirectional fusion feature. .

4. The method for bidirectional fusion image style transfer based on content and style decoupling according to claim 2, characterized in that: For the acquired style image set, feature extraction is performed to obtain the common style features , enhanced by self-attention, and the enhanced style common features are obtained The method comprises the following steps: Extracting common style features from a collection of style images , and perform weighted average to obtain the common features of the style the expression; Use the self-attention mechanism to integrate the common features of style The representation of is encoded as a style query vector, a style key vector, and a style value vector; According to the attention map The attention weights extracted from the style value vector are weighted to obtain the attention weighted average of the style features. ; According to the calculated attention map, style value vector and attention weighted average , calculate the attention weighted standard deviation ; Attention-weighted standard deviation and average , the common characteristics of style Enhance and obtain the common features of the enhanced style .

5. The method for bidirectional fusion image style transfer based on content and style decoupling according to claim 1, characterized in that: The method extracts text features based on the acquired text description, concatenates them with bidirectional fusion features to generate a conditional control vector as guidance information for image generation, performs a denoising inverse diffusion process guided by the conditional control vector on the acquired target image, and finally decodes and obtains a stylized image, including the following steps: Use a text encoder to extract text features from the acquired text description to obtain a text embedding vector; Fusion of text embedding vectors and bidirectional features After concatenation and fusion, the data is fed into the Text Transformer model to generate a conditional control vector. The denoising inverse diffusion process guided by the conditional control vector is performed on the acquired target image to obtain the final latent variable ; The final latent variable , decoded by the decoder to obtain a stylized image .

6. The method for bidirectional fusion image style transfer based on content and style decoupling according to claim 5, characterized in that: The denoising inverse diffusion process guided by the conditional control vector is performed on the bidirectional fusion features to obtain the final latent variable , including the following steps: The ReNoise strategy is used to gradually add noise to the target image, and then the noise is predicted by inverse diffusion denoising, and the predicted noise is added to the target image to obtain a noisy image. ; The noisy image Input into the Denoising U-Net module, and perform step-by-step denoising based on the conditional vector guidance to obtain the final latent variable .

7. The method for bidirectional fusion image style transfer based on content and style decoupling according to claim 1, characterized in that: It also includes the construction of a style transfer network implementation, including a bidirectional fusion module for common style enhancement and an SDM diffusion model. The bidirectional fusion module for common style enhancement includes a content extractor, a style extractor, and an SBF module. The content extractor is used to extract content features from the content image; the style extractor is used to extract style features; The SBF module is used to perform bidirectional fusion of style features and content features.

8. The method for bidirectional fusion image style transfer based on content and style decoupling according to claim 7, characterized in that: The style transfer network is trained using a contrast optimization strategy based on the decoupling of content and style features. The training process includes the following steps: Get the style image, content image, and target image after style transfer , build a training image set; Using the Style Extractor Extract the target image after style transfer Style characteristics , and the style characteristics of the content image ; Using content extractor Extract the target image after style transfer Content features in , and Content characteristics of style images ; Using the Style Extractor Extract the style features of style images of the same style and perform weighted average to obtain the common style features ; Using content extractor Extract content features of content images ; Construct style contrast learning and transfer the style to the target image Style characteristics Common features with style Construct a positive sample pair; the target image after style transfer Style characteristics Style features of content images Constructed as a negative sample pair; Construct content contrast learning to transfer the target image to the style Content Features and content characteristics Construct a positive sample pair; the target image after style transfer Content Features Content features of style images Constructed as a negative sample pair; Based on the constructed positive and negative sample pairs, comparative learning training is performed, and the parameters of the style transfer network are optimized based on the constructed overall loss function to obtain the trained style transfer network.

9. A bidirectional fusion image style transfer system based on content and style decoupling, characterized by: include: The bidirectional fusion module for common style enhancement is configured to obtain a set of style images and content images, extract common style features and content features respectively, and bidirectionally fuse the style features and content features using a cross-attention mechanism to generate a fused bidirectional fusion feature. The conditionally guided diffusion module is configured to extract text features based on the acquired text description, and concatenate them with the bidirectional fusion features to generate a conditional control vector as guidance information for image generation. The denoising inverse diffusion process guided by the conditional control vector is performed on the acquired target image, and finally a stylized image is decoded.

10. A bidirectional fusion image style transfer system based on content and style decoupling, characterized by: include: Image acquisition device and processor; The processor is configured to execute the steps of the bidirectional fusion image style transfer method based on content and style decoupling according to any one of claims 1 to 8.

Images