Bidirectional fusion image style transfer method and system based on content and style decoupling

Abstract

The application relates to the technical field of image conversion, and discloses a bidirectional fusion image style migration method and system based on content and style decoupling, which comprises the following steps: obtaining a style image set and a content image; respectively extracting common style features and content features; adopting a cross attention mechanism to bidirectionally fuse the style features and the content features to generate fused bidirectional fusion features; based on the obtained text description, extracting text features, splicing the text features and the bidirectional fusion features to generate a condition control vector as guide information for image generation; performing a denoising inverse diffusion process guided by the condition control vector on the obtained target image; and finally decoding to obtain a stylized image; based on the bidirectional feature fusion of the style and the content and the diffusion generation mechanism guided by the text, the image style migration effect with delicate style expression, clear semantic structure and strong controllability is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image conversion, in particular to a bidirectional fusion image style transfer method and system based on content and style decoupling. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the development of visual creative industries such as image editing, digital content creation and fashion design, style transfer technology is widely used to realize the fusion and reconstruction between image style and content. In particular, in the fashion design scene, the visual style such as brush strokes, textures and colors of the style image needs to be effectively transferred to the target image, while the content structure and semantic consistency of the target image are maintained, so as to realize the unity of style expression and content preservation. However, the style transfer task essentially involves cross-domain feature alignment and fusion, which requires capturing rich fine-grained style information and accurately preserving the content structure of the target image, which puts higher requirements on the feature modeling ability and fusion mechanism of the model.

[0004] Although current style transfer methods have achieved certain results in different application scenarios, they still face some problems when dealing with high complexity and high requirement image style transfer tasks: content and style fusion imbalance, feature entanglement, and insufficient style expression:

[0005] First of all, most methods use a one-way fusion strategy in feature fusion, with content features dominating style injection, ignoring the reverse effect of style features on content structure, resulting in unbalanced fusion results, content and style disconnection, and difficulty in achieving coordinated visual effects. For example, the generative adversarial network (GAN) method can realize image conversion without paired training data, but it has problems such as insufficient expression of style details, distortion of semantic structure, imbalance of content and style fusion, etc.

[0006] Secondly, whether it is a generative adversarial network (GAN) or a diffusion model, the phenomenon of "feature entanglement" often occurs during the migration process, that is, the style traces from the content image or the semantic structure from the style image are mixed into the output result, thereby affecting the semantic clarity and style purity of the image. For example, the method based on large-scale pre-training diffusion model (such as Stable Diffusion Model, SDM) can generate high-fidelity images while maintaining semantic consistency, thanks to its large number of parameters and massive training data. Related research further introduces text prompts as conditional control information to improve the artistic quality and controllability of generated images. Some people also propose a rich text editor to extract more detailed text prompts to constrain the content structure, thereby to some extent alleviating the problem of content structure drift. However, the above methods may still introduce style image content unrelated to the target content during style transfer, resulting in the destruction of the content structure.

[0007] Finally, due to the limited ability to extract and express fine-grained style information (such as brush strokes, textures, color matching, etc.), existing methods often have insufficient style expression in complex style image synthesis, making it difficult to achieve a refined and unified migration effect. These problems are particularly prominent in fashion design and other applications that require high precision in style, limiting the practicality and promotional value of style transfer technology. SUMMARY

[0008] To solve the above problems, the present application provides a bidirectional fusion image style transfer method and system based on content and style decoupling, which realizes the image style transfer effect with refined style expression, clear semantic structure and strong controllability based on the bidirectional feature fusion of style and content and the text-guided diffusion generation mechanism.

[0009] To achieve the above purpose, the present application adopts the following technical solutions:

[0010] One or more embodiments provide a bidirectional fusion image style transfer method based on content and style decoupling, comprising the following steps:

[0011] Obtain a set of style images and a content image, extract common style features and content features respectively, and use a cross-attention mechanism to bidirectionally fuse the style features and the content features to generate bidirectionally fused features;

[0012] Based on the obtained text description, extract text features and concatenate them with the bidirectionally fused features to generate a conditional control vector as the guiding information for image generation. Perform a denoising inverse diffusion process guided by the conditional control vector on the obtained target image, and finally decode to obtain a stylized image.

[0013] One or more embodiments provide a bidirectional fusion image style transfer system based on content and style decoupling, comprising:

[0014] The common style enhanced bidirectional fusion module is configured to obtain a style image set and a content image, extract common style features and content features respectively, perform bidirectional fusion on the style features and the content features by using a cross attention mechanism, and generate bidirectional fusion features after fusion;

[0015] The condition-guided diffusion module is configured to extract text features based on the obtained text description, splice the text features and the bidirectional fusion features to generate a condition control vector as guide information for image generation, perform a denoising inverse diffusion process guided by the condition control vector on the obtained target image, and finally decode to obtain a stylized image.

[0016] One or more embodiments provide a content and style decoupled bidirectional fusion image style transfer system, comprising:

[0017] An image acquisition device and a processor;

[0018] The processor is configured to perform the steps of the content and style decoupled bidirectional fusion image style transfer method described above.

[0019] Compared with the prior art, the present application has the following beneficial effects:

[0020] The present application realizes bidirectional feature fusion through a cross attention mechanism, improves the problem of incoordination between content and style fusion in traditional methods, makes the output image present more detailed style performance while retaining semantic structure, the introduction of the style image set improves the robustness of style feature extraction, avoids unstable style expression caused by individual sample differences, uses text features and bidirectional fusion features to construct a condition control vector, performs fine control of the image generation process at a semantic level, effectively suppresses the "feature entanglement" problem and improves the accuracy and controllability of style transfer, combines the step-by-step denoising property of the diffusion model, and makes the generated image have higher authenticity and detail integrity, and the overall scheme can be widely used in image generation scenarios with fine style requirements and clear semantics.

[0021] The advantages of the present application and the advantages of the additional aspects will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0022] The drawings accompanying the specification of the present application form a part thereof, serve to provide further understanding of the present application, and together with the exemplary embodiments of the present application and their description, serve to explain the present application, and do not constitute limitations thereof.

[0023] Figure 1 is the overall processing process flow schematic diagram of the content and style decoupled bidirectional fusion image style transfer of embodiment 1 of the present application;

[0024] Figure 2 is a structural schematic diagram of a common style enhanced bidirectional fusion module of embodiment 1 of the present application;

[0025] Figure 3 is a schematic diagram of a common style enhanced bidirectional fusion process of embodiment 1 of the present application;

[0026] Figure 4 is a comparison chart of ablation experiment results of embodiment 1 of the present application;

[0027] Figure 5 is a comparison chart of the influence of weight parameter ζ on the balance between content and style of embodiment 1 of the present application;

[0028] Figure 6 is a flowchart of a bidirectional fusion image style transfer method based on content and style decoupling of embodiment 1 of the present application; DETAILED DESCRIPTION

[0029] The present application will be further described below in conjunction with the drawings and embodiments.

[0030] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0031] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof. It should be noted that the various embodiments and features in the present application can be combined with each other without conflict. The embodiments will be described in detail below in conjunction with the drawings.

[0032] Embodiment 1

[0033] In the technical solutions disclosed in one or more embodiments, as shown in Figures 1 to 6 The bidirectional fusion image style transfer method based on content and style decoupling includes the following steps:

[0034] Step 1, obtaining a set of style images and content images, extracting common style features and content features respectively, using cross-attention mechanism to bidirectionally fuse style features and content features to generate fused bidirectional fusion features;

[0035] Step 2, based on the obtained text description, extract text features, splice with bidirectional fusion features to generate a condition control vector as the guide information for image generation, perform a denoising inverse diffusion process guided by the condition control vector on the obtained target image, and finally decode to obtain a stylized image;

[0036] The embodiment realizes bidirectional feature fusion through the cross attention mechanism, significantly improves the problem of inconsistent content and style fusion in traditional methods, makes the output image present more detailed style performance while retaining semantic structure, the introduction of the style image set improves the robustness of style feature extraction, avoids unstable style expression caused by individual sample differences, uses text features to construct a condition control vector for fine control of the image generation process at the semantic level, effectively suppresses the "feature entanglement" problem and improves the accuracy and controllability of style transfer, and combines the step-by-step denoising characteristics of the diffusion model to make the generated image have higher authenticity and detail integrity; the overall scheme can be widely used in image generation scenarios with fine style requirements and clear semantics;

[0037] In step 1, it is realized by constructing a common style enhanced bidirectional fusion module. First, bidirectional fusion is processed through the "style to content" and "content to style" bidirectional attention mechanism, effectively balancing content preservation and style transfer. Current methods often have difficulty capturing fine-grained details during complex style image transfer. Images of the same artistic style (such as Van Gogh style) have inherent common features; fusing these common features can result in more rich and representative style representations. Based on this, the embodiment designs a common style enhanced bidirectional fusion module to generate fused composite features;

[0038] The goal of the embodiment is to transfer the style of the style image to the content image and regenerate the content image; then the visual features (such as color, brush strokes, and texture) of the content image and the style features of the style image can be jointly transferred to the target image, which can be a fashion product image; for example, Figure 1 The example in the middle shows the image conversion and generation of a T-shirt image (fashion product image);

[0039] In step 1, the style image set and the content image are obtained, the common style feature representation and the content feature are extracted, and the cross attention mechanism is used to bidirectionally fuse the style feature representation and the content feature to generate the fused bidirectional fusion feature, as shown in Figure 2 and Figure 3 The generation method of the bidirectional fusion feature includes the following steps:

[0040] Step 11, for the obtained style image set, perform feature extraction to obtain style common features , enhanced by self-attention to obtain enhanced style common features ;

[0041] Step 111, extract style common features for the obtained style image set , and perform weighted average to obtain the representation of style common features ;

[0042] Images of the same style have inherent commonalities, such as Van Gogh style. Fusing them can produce more rich and representative style features. Therefore, in this embodiment, the style extractor extracts style features from a group of images with the same style , and then fuses them by weighted average to derive the representation of style common features , which are more rich and representative style features ;

[0043] Step 112, as shown in Figure 3 , encode the representation of style common features into style query vector , style key vector and style value vector using self-attention mechanism, to obtain attention map ;

[0044] (1);

[0045] wherein is a learnable parameter matrix; can be processed by normalization (Norm) and convolution operation (1×1Conv) before encoding;

[0046] The attention map can be calculated as:

[0047] (2);

[0048] wherein the superscript T represents transposition;

[0049] Step 113, weight the features in the style value vector according to the attention weights extracted from the attention map , to obtain the attention weighted average value of style features ;

[0050] The attention weighted average value is a representation that can highlight the most relevant style features, as shown below:

[0051] (3);

[0052] Step 114, according to the calculated attention map, style value vector and attention weighted average value , calculate attention weighted standard deviation ;

[0053] Specifically, in order to further enhance the representation of style features and reduce local distortion, the attention weighted standard deviation is derived in this embodiment to better capture the statistical local features of style features, as shown in the following method:

[0054] (4);

[0055] Step 115, based on the attention weighted standard deviation and average value , the style common features are enhanced to obtain enhanced style common features , as shown below:

[0056] (5);

[0057] Step 12, the enhanced style common features are encoded into enhanced style query vectors , style key vectors and style value vectors using self-attention;

[0058] Step 13, feature extraction is performed on the obtained content image, and the content query vector , content key vector and content value vector are encoded using attention mechanism.

[0059] Specifically, the , , and of the content features of the content image , key vector and value vector can be derived in the same way as described in formula (1).

[0060] Step 14, based on the obtained attention mechanism vector, cross attention mechanism is used for content to style and style to content bidirectional fusion to obtain bidirectional fusion features , including the following processes:

[0061] Step 141, content to style fusion: content query vector , enhanced style key vector and style value vector perform cross-attention operation (CA) to obtain the first fused feature

[0062] Fusion from content to style enables content features to guide style features. This process allows precise adjustment of style features according to content requirements, dynamically optimizing the representation of style features, thereby effectively enhancing the adaptability of the model to different content feature requirements. The fusion formula is as follows:

[0063] (6)

[0064] Step 142, fusion from style to content: perform cross-attention operation (CA) on the enhanced style query vector , the content key vector and the content value vector to obtain the second fused feature ;

[0065] One-way fusion lacks the step of style to content. Therefore, in order to balance content and style features, on the basis of the previous feature fusion guided by content features, this step uses the style query and the keys and values of content features to perform attention operation (CA). The focus of this stage is to adjust content features under the guidance of style features, and the formula is as follows:

[0066] (7)

[0067] Step 143, weighted fusion of the first fused feature and the second fused feature to obtain the bidirectional fused feature , as shown below:

[0068] (8)

[0069] wherein, is a weight parameter that balances the contributions of and .

[0070] This process can be expressed as:

[0071] ;

[0072] wherein, represents bidirectional fusion;

[0073] ​In the above implementation, to address the problem that existing methods struggle to capture fine-grained style information in complex style image transfer, and the inconsistency between content and style fusion caused by a single feature fusion strategy, a bidirectional fusion mechanism for enhancing common style is proposed. First, style features are extracted and fused from a set of images with the same style to construct a more representative and robust style commonality representation. Then, a weighted average and weighted standard deviation calculation method based on an attention mechanism is introduced to enhance the local expressive power of style features and improve the precision and consistency of style transfer. In the feature fusion stage, a bidirectional cross-attention strategy is adopted, employing both content-guided style and style-guided content, enabling content features and style features to achieve semantic coordination and complementarity, thereby solving problems such as weak style dominance and content structure distortion in traditional unidirectional fusion strategies. After processing by this bidirectional fusion mechanism, the model can achieve accurate fusion of style and content in terms of structure and visual representation, with complete and highly consistent style expression. The generated images are more visually harmonious, unified, and aesthetically consistent, showing significant advantages, especially in complex style transfer scenarios.

[0074] In step 2, the condition-guided inverse diffusion process involves extracting text features based on the acquired text description, concatenating them with bidirectional fusion features to generate a conditional control vector as guiding information for image generation, performing a denoising inverse diffusion process guided by the conditional control vector on the acquired target image, and finally decoding to obtain a stylized image. This method includes the following steps:

[0075] Step 21: Use a text encoder to extract text features from the acquired text description to obtain the text embedding vector. ;

[0076] like Figure 1 The natural language description shown is, for example, "a Van Gogh-style fashion piece with a blue sky and white cloud pattern"; the natural language description provides information on both the content "blue sky and white cloud pattern" and the style "Van Gogh style".

[0077] Optionally, the input text can be encoded using the CLIP Text Encoder to obtain a text embedding vector. ;

[0078] Step 22: Combine text embedding vectors with bidirectional fusion features After concatenation, the data is fed into the Text Transformer model to generate the conditional control vector C.

[0079] Step 23: Perform a denoising and inverse diffusion process guided by the conditional control vector on the acquired target image to obtain the final latent variables. It includes the following steps:

[0080] Step 231, gradually add noise to the target image using the ReNoise strategy, then predict the noise by inverse diffusion denoising, add the predicted noise to the target image to obtain a noise image ;

[0081] The gradual noise addition process is implemented by a noise adding module and is a forward process in the diffusion process.

[0082] The target image is a fashion single product image, and in this embodiment, a T-shirt image is taken as an example. The content image includes a blue sky and white cloud image and is converted into the style of Van Gogh, and finally a migrated target image is obtained.

[0083] Unlike traditional diffusion models, the starting point of the diffusion method used in this embodiment is to predict noise, rather than random noise.

[0084] Step 232, condition-guided denoising diffusion process: input the noise image into the Denoising U-Net module, and gradually denoise based on the condition vector to obtain the final latent variable ; This step is the reverse process in the diffusion process.

[0085] Denoising U-Net is a neural network used in the reverse stage of the diffusion process (denoising process). Its main function is to recover the original image from the noisy image, or at least predict the added noise in the image; including processes such as predicting noise, adding predicted noise for conditional diffusion, etc.

[0086] The Denoising U-Net module is based on the U-Net architecture, including:

[0087] Encoder (Encoder): Extract multi-scale features of the image, gradually downsample;

[0088] Decoder (Decoder): Restore high-level semantic information to image resolution, gradually upsample;

[0089] Skip connection (Skip Connections): Connect the encoder and decoder at the same level to preserve detailed information;

[0090] Time embedding module (Time Embedding): used to introduce the information of diffusion step t, to judge the stage of the current image denoising process;

[0091] Conditional module (Conditional Embedding): used to introduce condition information, such as category, text description, etc.

[0092] Step 24, image decoding: decoding the final latent variable , through a decoder (Decoder) to obtain a stylized image;

[0093] Further, the above noise adding module and the Denoising U-Net module constitute an SDM diffusion model; in the pre-trained SDM diffusion model, in order to better preserve the content of the fashion item in the back propagation process, the embodiment adopts a new inversion method named ReNoise. Specifically, this method first gradually adds noise to the fashion item image in the forward noise processing process to obtain a noisy image. Then, denoising is performed using the diffusion model, and the noise in the image is predicted in the reverse denoising process. The predicted noise as the initial latent signal is input into the SDM for further denoising. At the same time, in order to further enhance the content and style guidance, the embodiment generates a text embedding using a CLIP text encoder , and then combines it with to generate conditional control information through a text converter. Under the adjustment of the tensor , the diffusion process points to the desired style while preserving the original content.

[0094] Further, the optimization target of the SDM diffusion model in the training process is:

[0095] (9);

[0096] wherein, ; is fixed during training, indicating the operation of the neural network with frozen parameters; represents a content image, represents a style image; the target image after adding noise; Under the guidance of the conditional control information

[0097] , the trained SDM diffusion model generates a stylized image, i.e. the target image after style transfer .

[0098] In the above embodiments, by introducing the ReNoise inversion mechanism, noise is gradually added to the input fashion item image in the forward process to obtain a noise image under Gaussian disturbance; then, in the reverse denoising process, not only is the noise distribution predicted to restore the original content, but also the predicted initial latent signal is used as a key input to guide the subsequent generation process, so that the model can better maintain the semantic structure and visual features of the original image while reconstructing the style. Further, a conditional control tensor is generated based on text data to adjust the generation path of the entire diffusion process, so that it accurately points to the target style expression direction while maintaining content consistency. This method not only improves the model's response to the target style, but also enhances the consistency, interpretability and interactivity of image generation. Through the collaborative design of the ReNoise inversion and text condition control mechanisms, the proposed technical solution can achieve more accurate content preservation, more natural style transfer, and stronger generation controllability;

[0099] Further technical solutions, the process of steps 1 to 2 above is implemented by constructing a style transfer network, including a common style enhancement bidirectional fusion module and an SDM diffusion model, the common style enhancement bidirectional fusion module includes a content extractor, a style extractor and an SBF module;

[0100] The content extractor is used to extract content features from the content image; the style extractor is used to extract style features; and the SBF module is used to bidirectionally fuse the style features and the content features;

[0101] The SDM diffusion model is configured to extract text features based on the obtained text description, concatenate the text features with the bidirectionally fused features to generate a conditional control vector as guidance information for image generation, perform a denoising inverse diffusion process guided by the conditional control vector on the obtained target image, and finally decode to obtain a stylized image;

[0102] Further technical solutions also include a training process for the style transfer network using a contrast optimization strategy based on content and style feature decoupling, including the following steps:

[0103] Step S1, obtaining a set of training images including style images and content images, and target images after style transfer Step S2, feature extraction and decoupling: using a style extractor to extract style features of the target images after style transfer

[0104] and style features of the content images

[0105] Step S2, feature extraction and decoupling: using a style extractor to extract style features of the target images after style transfer and style features of the content images and style features of the content images ; using a content extractor extracting the content feature of the target image after style transfer , and the content feature of the first style image ;

[0106] the content feature of the target image after style transfer and the style feature of the target image after style transfer The extraction formula is:

[0107] (10);

[0108] Similarly, the style feature of the content image and the content feature of the first style image can be extracted, respectively, denoted as

[0109] (11);

[0110] Step S3, using a style extractor extracting the style feature of the style image of the same style, and performing weighted average to obtain the style common feature ; using a content extractor extracting the content feature of the content image ;

[0111] Step S4, constructing style contrast learning, constructing the style feature of the target image after style transfer and the style common feature as a positive sample pair; constructing the style feature of the target image after style transfer and the style feature of the content image as a negative sample pair ;

[0112] In this embodiment, the is a positive sample pair, and the is a negative sample pair. This stage can improve the ability of the model to identify the reference image style feature. The style consistency loss in the training process is:

[0113] (12);

[0114] wherein, is the first temperature coefficient. In a specific feature space, the distance between the target style feature and the style common feature of the style image can be reduced through contrast learning. ​​​​​

[0115] Step S5: Construct content contrast learning and transfer the style-transferred target image. Content features With content features Construct positive sample pairs; transfer the style-transferred target image Content features Content features of style images Construct negative sample pairs;

[0116] To minimize the content features of the target content and content image To account for the distance between them, a content contrast loss was designed. Treat them as positive sample pairs, These are considered negative sample pairs. The goal is to improve the model's ability to recognize content features of reference images. The content fidelity loss is constructed as follows:

[0117] (13);

[0118] in, It is the second temperature coefficient.

[0119] Step S5: Perform comparative learning training based on the constructed positive and negative sample pairs, and optimize the parameters of the style transfer network based on the constructed overall loss function to obtain the trained style transfer network.

[0120] Furthermore, the total loss includes losses related to condition control. Loss of style consistency and loss of content fidelity The total loss function is expressed as follows:

[0121] (14);

[0122] in, It is a hyperparameter that balances the contrast loss.

[0123] In traditional style transfer generation tasks, there is often representational entanglement between the style features of the content image and the content features of the style image, making it difficult for the model to effectively distinguish and extract pure content and style information, thus affecting the style consistency and structural fidelity of the generated image.

[0124] The contrast optimization strategy based on decoupling content and style features proposed in the above embodiments introduces a dual contrast learning mechanism during training, applying explicit contrast constraints to both content and style representations, thereby guiding the model to enhance its ability to separate and recognize the two types of features. This strategy significantly improves the model's decoupling expressive ability in the content and style modeling process, enabling the generated results to more accurately preserve the original content structure while achieving precise transfer of style features.

[0125] To verify the effectiveness and effect of the method described in this embodiment, a simulation experiment was conducted.

[0126] Ablation experiments were conducted to construct a variant of the style transfer network in this embodiment:

[0127] w / o-SBF variant: Replace the SBF module; to verify the validity of the SBF module, replace it with AdaAttN. Figure 4 (e) in the image shows the result of style transfer processing using the w / o-SBF variant;

[0128] w / o-Commonality variant: To verify the impact of the common style enhancement bidirectional fusion module on the model in this embodiment, a single style image is used instead of multiple style images for fashion style conversion; Figure 4 (d) in the image shows the result of style transfer processing using the w / o-Commonality variant;

[0129] w / o-Contrastive variant: To evaluate the importance of the contrastive optimization strategy in the model of this embodiment, the training method of the contrastive optimization strategy was removed. Figure 4 (f) in the image shows the result of style transfer processing using the w / o-Commonality variant;

[0130] Figure 4 In this example, (g) represents the image processed using the method described in this embodiment, employing the overall model of this embodiment, i.e. Figure 1 The complete style transfer network model was used, and a contrast optimization strategy based on the decoupling of content and style features was adopted to train the style transfer network model. The resulting image was obtained by performing style transfer on the image.

[0131] Figure 4 Image (a) in the image is in the style of Van Gogh. Figure 4 (b) in the image is the target image; the clothing image is used as the target image for the design. Figure 4 (c) in the image is the content image;

[0132] The method of this embodiment is superior to all variants, proving the effectiveness of each component in the method of this embodiment. Specifically: (1) Compared with the results generated by the original style transfer network of this embodiment, the stylized results of the w / o-SBF variant show more obvious artistic style, but the content preservation is relatively poor. This highlights the key role of the SBF module in achieving a harmonious balance between content and style. (2) The w / o-Commonality variant uses a single style image for conversion, but the generated image does not achieve the expected style effect. This shows that the scheme of this embodiment aggregates the style features of multiple images in the same style category, which can enhance the accurate capture and performance of style commonality. (3) The results generated by the w / o-Contrastive variant introduce patterns unrelated to the content image and styles deviating from the target style. These findings further confirm that the contrastive optimization strategy based on the decoupling of content and style features can enhance the model's ability to distinguish content and style.

[0133] In addition to the field of fashion style transfer, the method of this embodiment can also be applied to the field of artistic style transfer. To evaluate the performance of the style transfer network of this embodiment, the DIV2K dataset is used for the style transfer task.

[0134] In Figure 5 , the generated images are analyzed for changes in the parameter , which is a key factor in balancing and contribution, Content represents the content image. The increase in amplifies the artistic style, but also leads to the loss of image content, and vice versa. Figure 5 (a), (b), (c), and (d) in Figure 5 are the processing results of four content images (Content), respectively; for example, in (d) shows that when = 0.2, the appearance of the squirrel is very similar to the content image. In contrast, when = 1.0, the artistic style is very obvious, but the image content is significantly reduced. It is worth noting that although the latter has a strong artistic style, the outline of the squirrel is still preserved. In addition, it is observed that when = 0.6, the overall performance is best.

[0135] Embodiment 2

[0136] Based on Embodiment 1, a bidirectional fusion image style transfer system based on content and style decoupling is provided in this embodiment, which includes:

[0137] The common style enhanced bidirectional fusion module is configured to acquire a style image set and a content image, respectively extract common style features and content features, perform bidirectional fusion on the style features and the content features by using a cross attention mechanism, and generate bidirectional fusion features after fusion.

[0138] The condition-guided diffusion module is configured to extract text features based on the acquired text description, splice the text features and the bidirectional fusion features to generate a condition control vector as guide information for image generation, perform a denoising inverse diffusion process guided by the condition control vector on the acquired target image, and finally decode to obtain a stylized image.

[0139] It should be noted that each module in the embodiment corresponds to each step in Embodiment 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0140] Embodiment 3

[0141] Based on Embodiment 1, the embodiment provides a bidirectional fusion image style transfer system based on content and style decoupling, including:

[0142] An image acquisition device and a processor;

[0143] The image acquisition device is used to acquire style images and content images;

[0144] The processor is configured to perform the steps of the bidirectional fusion image style transfer method based on content and style decoupling described in Embodiment 1.

[0145] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0146] The above describes the specific embodiments of the present application in combination with the accompanying drawings, but is not a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications or changes made on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A bidirectional fusion image style transfer method based on content and style decoupling, characterized in that, Includes the following steps: Obtain a set of style images and content images, extract common style features and content features respectively, and use a cross-attention mechanism to perform bidirectional fusion of style features and content features to generate fused bidirectional features; Text features are extracted based on the acquired text description, and concatenated with bidirectional fusion features to generate a conditional control vector as guiding information for image generation. The acquired target image is then subjected to a denoising and inverse diffusion process guided by the conditional control vector, and finally the stylized image is decoded. The method for generating bidirectional fusion features includes the following steps: For the acquired set of style images, feature extraction is performed to obtain common style features. Enhancement is achieved through self-attention, resulting in common features of enhanced styles. ; This will enhance common stylistic features. Self-attention encoding is used to create enhanced style query vectors, style key vectors, and style value vectors; Feature extraction is performed on the acquired content images, and an attention mechanism is used to encode them into content query vectors, content key vectors, and content value vectors; Based on the obtained attention mechanism vectors, a cross-attention mechanism is used for bidirectional fusion from content to style and from style to content, resulting in bidirectional fused features. .

2. The bidirectional fusion image style transfer method based on content and style decoupling as described in claim 1, characterized in that: Based on the obtained attention mechanism vectors, a cross-attention mechanism is used for bidirectional fusion from content to style and from style to content, resulting in bidirectional fused features. This includes the following processes: Content query vector Enhanced style key vectors and style value vector Perform cross-attention operation to obtain the first fusion feature; Enhanced style query vector Content key vector and content value vector Perform cross-attention operation to obtain the second fusion feature; The first and second fusion features are weighted and fused to obtain the bidirectional fusion features. .

3. The bidirectional fusion image style transfer method based on content and style decoupling as described in claim 1, characterized in that: For the acquired set of style images, feature extraction is performed to obtain common style features. Enhancement is achieved through self-attention, resulting in common features of enhanced styles. The method includes the following steps: Extract common style features from the obtained style image set Weighted averages are used to obtain common style characteristics. The representation of; Employing a self-attention mechanism to identify common style features The style is represented as a style query vector, a style key vector, and a style value vector. According to attention map The extracted attention weights are used to weight the features within the style value vector, resulting in an attention-weighted average of the style features. ; Based on the calculated attention map, style value vector, and attention weighted average... Calculate the attention-weighted standard deviation ; Based on attention-weighted standard deviation and average Common characteristics of style Enhancement is performed to obtain common features of the enhanced style. .

4. The bidirectional fusion image style transfer method based on content and style decoupling as described in claim 1, characterized in that: The method involves extracting text features based on the acquired text description, concatenating them with bidirectional fusion features to generate a conditional control vector as guiding information for image generation, performing a denoising and inverse diffusion process guided by the conditional control vector on the acquired target image, and finally decoding to obtain a stylized image. The method includes the following steps: A text encoder is used to extract text features from the acquired text description to obtain a text embedding vector; Text embedding vectors and bidirectional fusion features After splicing and merging, the data is fed into the Text Transformer model to generate conditional control vectors; A denoising and inverse diffusion process guided by conditional control vectors is performed on the acquired target image to obtain the final latent variables. ; Final latent variables The stylized image is obtained by decoding using a decoder. .

5. The bidirectional fusion image style transfer method based on content and style decoupling as described in claim 4, characterized in that: A denoising inverse diffusion process guided by a conditional control vector is performed on the bidirectional fused features to obtain the final latent variables. This includes the following steps: A ReNoise strategy is used to progressively add noise to the target image. Then, inverse diffusion denoising is used to predict the noise, and the predicted noise is added to the target image to obtain a noisy image. ; Noise image The input is fed into the Denoising U-Net module, where conditional vector-guided stepwise denoising is performed to obtain the final latent variables. .

6. The bidirectional fusion image style transfer method based on content and style decoupling as described in claim 1, characterized in that: It also includes the construction of a style transfer network, including a bidirectional fusion module for common style enhancement and an SDM diffusion model. The bidirectional fusion module for common style enhancement includes a content extractor, a style extractor, and an SBF module. The content extractor is used to extract content features from a content image; the style extractor is used to extract style features. The SBF module is used for bidirectional fusion of style features and content features.

7. The bidirectional fusion image style transfer method based on content and style decoupling as described in claim 6, characterized in that: The style transfer network is trained using a contrastive optimization strategy based on the decoupling of content and style features. The process includes the following steps: Acquire the style image, content image, and the target image after style transfer. Construct a training image set; Use style extractor Extracting the target image after style transfer stylistic features and the stylistic features of the content images Use content extractor Extracting the target image after style transfer Content features and the Content characteristics of style images ; Use style extractor Extract style features from style images of similar styles, and perform a weighted average to obtain common style features. Use content extractor Extracting content features from content images ; Constructing style contrast learning to transfer style to the target image stylistic features Common characteristics of style Construct positive sample pairs; transfer the style-transferred target image stylistic features Stylistic features of content images Construct negative sample pairs; Construct content-based contrastive learning to transfer style to the target image. Content features With content features Construct positive sample pairs; transfer the style-transferred target image Content features Content features of style images Construct negative sample pairs; The style transfer network is trained by comparative learning based on the constructed positive and negative sample pairs, and the parameters of the style transfer network are optimized based on the constructed overall loss function to obtain the trained style transfer network.

8. A bidirectional fusion image style transfer system based on content and style decoupling, characterized in that, include: The common style enhancement bidirectional fusion module is configured to acquire a set of style images and content images, extract common style features and content features respectively, and use a cross-attention mechanism to perform bidirectional fusion of style features and content features to generate fused bidirectional fusion features; The condition-guided diffusion module is configured to extract text features based on the acquired text description, concatenate them with bidirectional fusion features to generate a conditional control vector as guiding information for image generation, perform a denoising inverse diffusion process guided by the conditional control vector on the acquired target image, and finally decode to obtain a stylized image. The method for generating bidirectional fusion features includes the following steps: For the acquired set of style images, feature extraction is performed to obtain common style features. Enhancement is achieved through self-attention, resulting in common features of enhanced styles. ; This will enhance common stylistic features. Self-attention encoding is used to create enhanced style query vectors, style key vectors, and style value vectors; Feature extraction is performed on the acquired content images, and an attention mechanism is used to encode them into content query vectors, content key vectors, and content value vectors; Based on the obtained attention mechanism vectors, a cross-attention mechanism is used for bidirectional fusion from content to style and from style to content, resulting in bidirectional fused features. .

9. A bidirectional fusion image style transfer system based on content and style decoupling, characterized in that, include: Image acquisition device and processor; The processor is configured to perform the steps of the bidirectional fusion image style transfer method based on content and style decoupling as described in any one of claims 1-7.

Images