Text-driven image style migration method and system based on multilevel feature fusion and gating

Abstract

The invention relates to a text-driven image style migration method and system based on multilevel feature fusion and gating. The method comprises the following steps: inputting a content image into a visual geometry group image encoder to obtain image features, and inputting text description into a contrast language image pre-training text encoder to obtain text features; inputting the image features into a texture semantic self-adaptive gating module to generate texture semantic fusion image features, and inputting the text features into a multi-modal feature coordinator to generate modulation style text features; inputting the texture semantic fusion image features and the modulation style text features into an image text hierarchy fusion network module to obtain image migration features; and inputting the image migration features into a decoder to generate a style image. By adopting the method, the time and space cost can be reduced, the visual quality of image style migration is improved, the content structure retention and style migration effects are considered, and cross-level feature fusion and multi-granularity style decoupling are realized.

Description

Technical Field

[0001] This application belongs to the field of image style transfer, and in particular relates to a text-driven image style transfer method and system based on multi-level feature fusion and gating. Background Technology

[0002] Style transfer refers to the process of re-rendering a content image using a given target style. Traditional methods typically utilize style images to represent the target style and have achieved significant success in terms of quality, efficiency, and generalization ability. However, these methods require pre-prepared style images and are limited by existing specific artistic styles, thus restricting their flexibility and creativity.

[0003] Recently, some studies have focused on addressing this issue by replacing stylistic images with text prompts. The core motivation is that language is the most natural and widely used medium of communication for humans, enabling users to describe the desired style in a more flexible way and to use their imagination to create more creative and abstract art-style images.

[0004] However, current text-driven style transfer methods are still in their early stages and face many challenges. For example, some text-driven style transfer methods require separate model training for each style, resulting in high time and space costs, and the generated images are prone to artifacts and distortions. Other text-driven style transfer methods rely on a large amount of image-text pair training data, which is difficult and costly to collect, but the transfer effect is coarse. In addition, the networks used to extract content features and the networks used to extract text features in some text-driven style transfer methods have significant differences in feature space due to different pre-training targets and datasets. Direct fusion will lead to modality mismatch and affect the style transfer effect. Furthermore, existing style transfer methods also have two other problems: First, existing methods have hierarchical fragmentation, that is, there is a separation between low-level information (edges / texture) and high-level information (objects / semantics) in the generated images. Second, existing technologies lack dynamic control, that is, using a fixed fusion strategy cannot adapt to diverse style granularities (e.g., "watercolor" requires global granular blending, while "embossing" requires local granular highlights). Summary of the Invention

[0005] Therefore, it is necessary to address the aforementioned technical issues by providing a text-driven image style transfer method and system based on multi-level feature fusion and gating, which can improve the visual quality of image style transfer results without requiring separate model training for each style cue, balance content structure preservation and style transfer effect, and achieve cross-level feature fusion and multi-granularity style decoupling.

[0006] This application provides a text-driven image style transfer method based on multi-level feature fusion and gating, including: The content image and text description are obtained. The content image is input into the visual geometry group image encoder to obtain image features, and the text description is input into the contrastive language image pre-trained text encoder to obtain text features. Image features are input into the texture semantic adaptive gating module to generate texture semantic fusion image features, and text features are input into a multimodal feature coordinator built on a fully connected layer to generate modulation style text features; The image features, modulation style text features, and image features are input into the image-text hierarchical fusion network module to obtain image transfer features; Image transfer features are input into a decoder built on a convolutional neural network to generate style images, which are used to characterize the image style transfer results of content images under the guidance of text descriptions.

[0007] The aforementioned text-driven image style transfer method and system based on multi-level feature fusion and gating extracts image features of the content image through a Visual Geometric Group (VGG) image encoder, and extracts text features of the text description through a Contrastive Language Image Pre-trained (CLIP) text encoder. This enables accurate extraction of the spatial structure of the content image and the semantic style of the text, thus simultaneously preserving image structure and capturing text semantics, providing high-quality basic features for subsequent cross-modal style transfer.

[0008] Furthermore, the aforementioned text-driven image style transfer method and system based on multi-level feature fusion and gating processes image features through a texture semantic adaptive gating module. This allows for dynamic adjustment of the fusion ratio between texture details and semantic structure in the image according to text style requirements, thereby avoiding structural distortion and detail loss in the stylized result caused by texture and semantic imbalance. Consequently, it generates texture semantic fusion image features that possess both structural integrity and stylistic detail, thus enabling cross-level feature fusion. By modulating text features through a multi-modal feature coordinator based on fully connected layers, it transforms text features in the CLIP feature space into modulated style text features compatible with the VGG image feature space. This avoids semantic misalignment in cross-modal fusion, thereby effectively transmitting text style information to image features.

[0009] Furthermore, existing fusion processors (such as UNITER's single-stream Transformer) treat all features equally, ignoring the differentiated requirements of shallow (texture), mid-level (object), and deep (semantics) layers in style transfer. The aforementioned text-driven image style transfer method and system based on multi-level feature fusion and gating, through an image-text hierarchical fusion network module, fuses texture-semantic fusion image features and modulated style text features. This enables hierarchical alignment of text description and style granularity, thereby solving the problem of insufficient hierarchical interaction caused by the lack of style hierarchy modeling in traditional fusion methods. Consequently, it can generate image transfer features with both rich detail and overall consistency, achieving multi-granularity style decoupling and providing a precise control basis for text-guided image style transfer.

[0010] Furthermore, the above-mentioned text-driven image style transfer method and system based on multi-level feature fusion and gating processes the image transfer features through a decoder built on a convolutional neural network, which can restore the abstract image transfer features into a visually coherent and clear style image, thereby achieving high-quality style transfer guided by text. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a text-driven image style transfer method based on multi-level feature fusion and gating, provided as an embodiment of this application. Figure 1 ; Figure 2 A schematic diagram of the structure of a text-guided style transfer framework provided in one embodiment of this application; Figure 3 A flowchart illustrating a text-driven image style transfer method based on multi-level feature fusion and gating, provided as an embodiment of this application. Figure 2 ; Figure 4 A schematic diagram of the structure of an image-text hierarchical fusion network module provided in one embodiment of this application; Figure 5 This is a schematic diagram of the structure of a text semantic style injection submodule provided in one embodiment of this application; Figure 6 This is a schematic diagram illustrating a process for generating texture semantic fusion image features based on image features, as provided in one embodiment of this application. Figure 7 This is a schematic diagram of the structure of a texture semantic adaptive gating module provided in one embodiment of this application; Figure 8 This is a schematic diagram of the structure of a multi-scale fusion submodule provided in one embodiment of this application; Figure 9 A schematic diagram of the structure of a feature enhancement submodule provided in one embodiment of this application; Figure 10 A schematic diagram illustrating the structure of a text-guided style transfer framework during training, provided in one embodiment of this application; Figure 11 This is a schematic diagram of the structure of a text-driven image style transfer system based on multi-level feature fusion and gating, provided as an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0014] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0015] As an illustration, the Visual Geometry Group (VGG) is a deep convolutional neural network proposed by the Visual Geometry Group at Oxford University. Increasing network depth can improve model performance. Its network structure is simple, regular, and highly scalable. Common versions include VGG11, VGG13, VGG16, and VGG19. In this paper, VGG can be used as an image encoder, mapping content images to a specific feature space, which helps preserve the content structure of the image.

[0016] Optionally, Contrastive Language-Image Pretraining (CLIP), a multimodal model proposed by OpenAI in 2021, is trained based on contrastive learning of text-image pairs. It can learn image-text matching relationships and perform tasks such as image-text feature similarity calculation, cross-modal retrieval, and zero-shot image classification. In this paper, CLIP's text encoder can be used to extract text style features, and CLIP's image encoder can be used to evaluate the semantic consistency between the generated image and the target text.

[0017] In one exemplary embodiment of this application, please refer to Figure 1 and Figure 2This paper provides a text-driven image style transfer method based on multi-level feature fusion and gating. This embodiment illustrates the application of this method to an image transfer terminal. It is understood that this method can also be applied to an image transfer server, and further to an image transfer system including both an image transfer terminal and an image transfer server, implemented through the interaction between the two. In this embodiment, the text-driven image style transfer method based on multi-level feature fusion and gating includes the following steps: Step S101: Obtain the content image and text description. Input the content image into the visual geometry group image encoder to obtain image features, and input the text description into the contrastive language image pre-trained text encoder to obtain text features.

[0018] Specifically, the image transfer terminal can acquire content images and text descriptions, and input the content image into a Visual Geometric Group (VGG) image encoder to map the content image to the VGG feature space, obtaining image features. The image transfer terminal can also input the text description into a Contrastive Language Image Pre-trained (CLIP) text encoder, mapping the text description to the CLIP feature space, obtaining text features. The content image can be the original image to be style transferred, containing the content structure and semantic information to be preserved. The image features can be multi-scale high-dimensional feature vectors extracted through the VGG network, containing texture, edge, contour, and semantic information, which can be used for subsequent feature fusion and style transfer. The text description can be a natural language style description, which can be used to guide image style transfer. The text features can be used to characterize the style attributes and abstract concepts of the text description.

[0019] Preferably, the VGG image encoder can be constructed based on the VGG19 model.

[0020] Optionally, the VGG image encoder and CLIP text encoder can be installed on the image migration terminal, the image migration server, or an image migration system that includes both the image migration terminal and the image migration server; no limitation is made here.

[0021] To illustrate, the VGG image encoder excels at preserving content information such as spatial layout and object structure in images, while the CLIP text encoder can transform abstract text style descriptions into computable semantic features. Therefore, the VGG image encoder, leveraging its pre-trained feature extraction capabilities, can accurately extract the core structural features of content images; the CLIP text encoder, with its advantages in cross-modal semantic understanding, can accurately capture style attributes in text descriptions, providing clear semantic guidance for style transfer.

[0022] In step S102, the image features are input into the texture semantic adaptive gating module to generate texture semantic fusion image features, and the text features are input into the multimodal feature coordinator built based on the fully connected layer to generate modulation style text features.

[0023] Specifically, the image transfer terminal can input image features from different network levels generated by the VGG image encoder into the texture semantic adaptive gating module for multi-scale feature fusion, generating texture semantic fused image features. It can also input text features generated by the CLIP text encoder into a multimodal feature coordinator built on fully connected layers, mapping the text features in the CLIP feature space to the VGG feature space to generate modulation-style text features. Lower network level image features can contain details such as texture and edges, mid-level network level features can contain shape and local structural information, and higher network level features can contain semantic and global structural information.

[0024] Optionally, a Multi-Modal Feature Harmonizer (MMFH) can be used to address the modality mismatch problem caused by the gap between the CLIP feature space and the VGG feature space. The MMFH can modulate and expand the dimensionality of the style features extracted from the CLIP feature space of the text description to better fuse with the content features of the VGG feature space.

[0025] Optionally, the number of fully connected layers in the MMFH can preferably be three.

[0026] For example, the Semantic-Texture Adaptive Gating Module (STAG) aims to solve the semantic fragmentation problem when content images are extracted with multi-scale features by a VGG image encoder. STAG can receive feature inputs at different levels and enhance feature complementarity through adaptive fusion. It can adaptively adjust the feature fusion strategy according to the style type of the text prompt, providing a balanced content representation for subsequent cross-modal interactions.

[0027] As an illustration, multi-scale features extracted from content images by a VGG image encoder often face the problem of semantic fragmentation: deep, high-dimensional features may lose texture details, while shallow, low-dimensional features lack semantic context, leading to structural distortion or style-content mismatch in the stylization results. Therefore, in this embodiment, multi-scale feature fusion can be performed using STAG to generate texture-semantic fusion image features, thereby avoiding the potential disconnect between deep semantics and shallow texture.

[0028] Optionally, STAG and MMFH can be installed on an image migration terminal, an image migration server, or an image migration system that includes both an image migration terminal and an image migration server; no limitation is made here.

[0029] Step S103: Input the texture semantic fusion image features, modulation style text features and image features into the image-text hierarchical fusion network module to obtain image transfer features.

[0030] Specifically, the image transfer terminal can input the texture semantic fusion image features generated by STAG, the modulation style text features generated by MMFH, and the image features generated by the VGG image encoder into the image-text hierarchical fusion network module to obtain image transfer features. These image transfer features can be used to characterize the comprehensive information features generated after fusion, encompassing both content fidelity and style transfer.

[0031] Optionally, the Image-Text Hierarchical Fusion Network (ITHFN) can achieve accurate mapping and fusion of different style levels in texture semantic fusion image features and modulated style text features, enhance fusion features, and promote the interaction of image and text fusion features at different depth levels, thereby improving the overall effect of style transfer.

[0032] Optionally, ITHFN can be installed on an image migration terminal, an image migration server, or an image migration system that includes both an image migration terminal and an image migration server; no limitation is made here.

[0033] Step S104: Input the image transfer features into the decoder built on the convolutional neural network to generate a style image.

[0034] Specifically, the image transfer terminal can input the image transfer features generated by ITHFN into a decoder built on a convolutional neural network to generate a style image.

[0035] Optionally, a style image can be used to characterize the image style transfer results of the content image guided by the text description.

[0036] Optionally, the decoder can be constructed from a multi-layer convolutional neural network.

[0037] Indicatively, the decoder can gradually recover the spatial resolution and detail information of an image through a convolutional neural network, transforming the fused style and content features into a visual image, ensuring that the transferred style can be presented naturally, and ultimately achieving high-quality style transfer guided by text.

[0038] In the aforementioned text-driven image style transfer method based on multi-level feature fusion and gating, the VGG image encoder extracts image features, which can fully preserve the spatial structure of the content image and ensure the integrity of the main content after style transfer. The CLIP text encoder extracts text features, which can accurately capture the style semantics in the text description and provide accurate directional guidance for style transfer. The STAG image feature processing addresses the feature fragmentation of VGG by bridging shallow texture and deep semantics based on a gating mechanism, which can alleviate the problem of the disconnect between deep semantics and shallow texture, realize cross-level feature fusion, and improve the visual quality of style images. The MMFH based on fully connected layers can effectively reconcile the feature space differences between VGG and CLIP, thereby solving the modality mismatch problem caused by direct fusion of cross-modal features and making the combination of text style and image content more natural. The ITHFN module performs deep fusion of texture semantic fusion image features and modulated style text features, which can realize hierarchical transfer and adaptive fusion of style features.

[0039] Furthermore, the text-driven image style transfer method based on multi-level feature fusion and gating described above does not require training a separate model for each text prompt, nor does it rely on a large amount of paired training data. This reduces time and space costs while improving the flexibility and applicability of style transfer.

[0040] In an optional embodiment of this application, please refer to Figure 1 , Figure 3 and Figure 4 The ITHFN module may include a text semantic style injection submodule, a fusion feature enhancement submodule, and a hierarchical feature interaction submodule. In step S103, the texture semantic fusion image features, modulation style text features, and image features are input into the image-text hierarchical fusion network module to obtain image transfer features, which may include: Step S303: Input the modulation style text features and texture semantic fusion image features into the text semantic style injection submodule of the image text hierarchical fusion network module to generate text image fusion features.

[0041] Optionally, the text semantic style injection submodule may include, but is not limited to, a fine text semantic style injection submodule, a medium text semantic style injection submodule, and a coarse text semantic style injection submodule, and the text image fusion features may include, but are not limited to, fine-grained text image fusion features, medium-grained text image fusion features, and coarse-grained text image fusion features.

[0042] Specifically, the image transfer terminal can input modulation style text features and texture semantic fusion image features into the coarse text semantic style injection submodule of ITHFN to generate coarse-grained text image fusion features; the image transfer terminal can input coarse-grained text image fusion features and texture semantic fusion image features into the medium text semantic style injection submodule of ITHFN to generate medium-grained text image fusion features; the image transfer terminal can input medium-grained text image fusion features and texture semantic fusion image features into the fine text semantic style injection submodule of ITHFN to generate fine-grained text image fusion features.

[0043] Optionally, fine-grained text-image fusion features may be used, but are not limited to, text-image fusion features that describe edge textures and local brushstrokes; medium-grained text-image fusion features may be used, but are not limited to, text-image fusion features that describe object shapes and local layouts; and coarse-grained text-image fusion features may be used, but are not limited to, text-image fusion features that describe overall tone and global style.

[0044] As an illustration, style attributes in images typically exhibit a multi-scale distribution. However, traditional Transformer models lack the ability to model differences in style granularity, leading to inaccurate alignment between text descriptions and visual styles. This makes semantic distortion and loss of detail prone to occur during style transfer. To address this issue, a multi-granularity style-aware cross-modal Transformer can be used. This transformer overcomes the limitations of homogeneous Transformers by leveraging hierarchical cross-attention, aligning text descriptions with style granularity at different levels (e.g., Coarse-level captures the global characteristics of "oil painting," while Fine-level resolves local details of "coarse brushstrokes"). This provides a precise control foundation for text-guided style transfer.

[0045] Optionally, the Text Semantic Style Injection Module (TeSSI) can fuse modulation style text features and texture semantic fusion image features through an attention mechanism.

[0046] Step S304: Input each text image fusion feature into the fusion feature enhancement submodule of the corresponding image-text hierarchical fusion network module to generate text image enhancement fusion features corresponding to each text image fusion feature.

[0047] Specifically, the image migration terminal can input the text image fusion features generated by each TeSSI into the fusion feature enhancement submodule of the ITHFN module corresponding to each TeSSI, and generate the text image enhancement fusion features corresponding to each text image fusion feature.

[0048] Optionally, the Fusion Feature Enhancement (FFE) submodule can independently enhance the representation of the fused text-image features at different depth levels.

[0049] Optionally, the expression for the FFE submodule can be:

[0050] In the formula, For particle size number Text image enhancement fusion features, For particle size number Text-image fusion features, For particle size number The multi-head self-attention layer of the FFE submodule, For particle size number The layer normalization layer of the FFE submodule.

[0051] Optionally, TeSSI may include fine TeSSI (such as...) Figure 4 (e.g., fine text semantic style injection submodule), TeSSI (e.g.) Figure 4 The Chinese text semantic style injection submodule) and coarse TeSSI (such as Figure 4 The coarse text semantic style injection submodule in the text includes corresponding fine-grained, medium-grained, and coarse-grained FFEs. Text image enhancement fusion features include fine-grained, medium-grained, and coarse-grained text image enhancement fusion features. Text image fusion features include fine-grained, medium-grained, and coarse-grained text image fusion features. For example, the granularity number... It can belong to When the particle size number At that time, TeSSI can be fine TeSSI, and FFE can be the corresponding fine-grained FFE. This can be a multi-head self-attention layer for fine-grained FFE, enhancing text and image fusion features. It can enhance the fusion features of fine-grained text images, text image fusion features It can be used for fine-grained text-image fusion features; when the granularity number At that time, TeSSI can be medium-sized TeSSI, and FFE can be the corresponding medium-grained FFE. This can be a multi-head self-attention layer for medium-granularity FFE, enhancing text and image fusion features. Enhanced fusion features can be provided for medium-granular text images; text-image fusion features It can be a medium-granularity text-image fusion feature; when the granularity number In this case, TeSSI can be coarse TeSSI, and FFE can be the corresponding coarse-grained FFE. This can be a multi-head self-attention layer for coarse-grained FFE, enhancing text and image fusion features. It can enhance the fusion features of coarse-grained text images, text image fusion features It can be used for coarse-grained text-image fusion features.

[0052] Step S305: Input each text image enhancement fusion feature into the hierarchical feature interaction submodule of the image-text hierarchical fusion network module to obtain text image hierarchical fusion features.

[0053] Specifically, the migration terminal can input the text image enhancement and fusion features generated by each FFE into the hierarchical feature interaction submodule of the ITHFN module to obtain the text image hierarchical fusion features.

[0054] Optionally, taking text image enhancement fusion features including fine-grained text image enhancement fusion features, medium-grained text image enhancement fusion features, and coarse-grained text image enhancement fusion features as an example: text image hierarchical fusion features can be used to characterize the multi-level adaptive fusion result of multi-level adaptive fusion of fine-grained text image fusion features, medium-grained text image fusion features, and coarse-grained text image fusion features.

[0055] Furthermore, the hierarchical feature interaction submodule can use a cross-attention mechanism to perform feature interaction on fine-grained text image enhancement fusion features, medium-grained text image enhancement fusion features, and coarse-grained text image enhancement fusion features to obtain fine-grained interactive text image enhancement fusion features, medium-grained interactive text image enhancement fusion features, and coarse-grained interactive text image enhancement fusion features. Then, it can fuse these fine-grained interactive text image enhancement fusion features, medium-grained interactive text image enhancement fusion features, and coarse-grained interactive text image enhancement fusion features to generate hierarchical text image fusion features.

[0056] As an illustration, the Hierarchical Feature Interaction (HFI) submodule can focus on cross-attention computation between different depth levels, allowing image-text fusion features at different abstraction levels to guide each other: shallow fine-grained features can obtain global semantic guidance by interacting with deep coarse-grained features, and deep coarse-grained features can obtain detail enhancement by interacting with shallow fine-grained features, thereby achieving hierarchical transfer and adaptive fusion of style features.

[0057] For example, taking text image enhancement fusion features, including fine-grained, medium-grained, and coarse-grained text image enhancement fusion features, as an example, the expression for the cross-attention mechanism of the hierarchical feature interaction submodule can be:

[0058]

[0059]

[0060] In the formula, , and These are fine-grained interactive text-image enhancement and fusion features, medium-grained interactive text-image enhancement and fusion features, and coarse-grained interactive text-image enhancement and fusion features, respectively. , and These are fine-grained text-image enhancement and fusion features, medium-grained text-image enhancement and fusion features, and coarse-grained text-image enhancement and fusion features, respectively. , and These are cross-attention layers corresponding to fine-grained text image enhancement and fusion features, medium-grained text image enhancement and fusion features, and coarse-grained text image enhancement and fusion features, respectively. This is a layer normalization layer. At this point, the query vector corresponds to the cross-attention layer of the fine-grained text-image enhancement fusion feature. It can be Key vector It can be And value vector It can be ; The query vector corresponding to the cross-attention layer of medium-granularity text-image enhancement fusion features It can be Key vector It can be And value vector It can be ; The query vector corresponding to the cross-attention layer of medium-granularity text-image enhancement fusion features It can be Key vector It can be And value vector It can be .

[0061] Step S306: After concatenating the modulation style text features and text image hierarchical fusion features, perform feature fusion with the image features to generate image transfer features.

[0062] Specifically, such as Figure 4 As shown, the image migration terminal can perform feature splicing on modulation style text features and text-image hierarchical fusion features to obtain image-text splicing features, and then fuse the image-text splicing features and image features to generate image migration features.

[0063] In the aforementioned text-driven image style transfer method based on multi-level feature fusion and gating, the TeSSI module in the ITHFN module performs hierarchical injection of modulated style text features and texture semantic fusion image features, enabling accurate mapping of text style information at different visual granularities. The FFE submodule enhances the text-image fusion features at each granularity, strengthening the correlation of features within the same granularity through a multi-head self-attention mechanism, thereby generating text-image enhancement fusion features with clearer representation and stronger consistency. The hierarchical feature interaction module enables cross-level interaction of text-image enhancement fusion features at each granularity, achieving guidance and supplementation between features of different granularities, thus generating text-image hierarchical fusion features with multi-level information. By performing feature splicing and fusing image features, the method can enhance global style consistency while preserving multi-level style details, constraining the matching direction of text style and image content as a whole, thereby generating image transfer features that balance detail richness and overall coordination.

[0064] In an optional embodiment of this application, please refer to Figure 5 The text semantic style features and texture semantic fusion image features are input into the text semantic style injection submodule of the image-text hierarchical fusion network module to generate text-image fusion features, which may include: Specifically, the image transfer terminal can input the texture semantic fusion image features into the multi-head self-attention layers of coarse TeSSI, medium TeSSI, and fine TeSSI, and obtain the texture semantic fusion image attention features after passing through the residual connection layer and the normalization layer.

[0065] Specifically, the image transfer terminal can input modulation style text features and texture semantic fusion image attention features into the multi-head cross attention layer of coarse TeSSI, and after passing through the residual connection layer and normalization layer, generate coarse-grained text image preliminary fusion features.

[0066] Specifically, the image transfer terminal can input the preliminary fusion features of the text image into the multilayer perceptron of TeSSI to obtain coarse-grained text image fusion features.

[0067] Specifically, the image transfer terminal can input coarse-grained text image fusion features and texture semantic fusion image attention features into the multi-head cross-attention layer of the TeSSI. After passing through the residual connection layer and the normalization layer, it generates preliminary medium-grained text image fusion features, and inputs the preliminary medium-grained text image fusion features into the multilayer perceptron of the TeSSI to obtain the medium-grained text image fusion features. Specifically, the image transfer terminal can input the medium-granularity text image fusion features and texture semantic fusion image attention features into the multi-head cross-attention layer of fine TeSSI. After passing through the residual connection layer and normalization layer, the fine-granularity text image preliminary fusion features are generated. The fine-granularity text image preliminary fusion features are then input into the multilayer perceptron of fine TeSSI to obtain the fine-granularity text image fusion features.

[0068] Optionally, the expression for the multi-head cross-attention layer can be:

[0069] In the formula, For multi-head cross-attention layer, , and These are the query vector based on texture-semantic fusion image attention features, the key vector and value vector based on modulation style text features, coarse-grained text-image fusion features, or medium-grained text-image fusion features, respectively. For activation function, This is the transpose of the matrix. The dimension of the multi-head cross-attention layer.

[0070] In the above-mentioned text-driven image style transfer method based on multi-level feature fusion and gating, the multi-head self-attention layer of TeSSI can enhance the correlation within the texture semantic fusion image features; and the multi-head cross-attention layer can interact with the modulation style text features and the texture semantic fusion image attention features to achieve accurate correlation between text style features and image content features.

[0071] In an optional embodiment of this application, the multilayer perceptron of the text semantic style injection submodule can be replaced with a Kolmogorov-Arnold network.

[0072] Alternatively, the Kolmogorov-Arnold Network (KAN) is a neural network architecture inspired by the Kolmogorov-Arnold representation theorem. The Kolmogorov-Arnold representation theorem states that any multivariable continuous function defined on a bounded domain can be represented as a combination and addition of a finite number of univariate continuous functions. KAN concretizes the "univariate function" in the theorem into a learnable curve, thus replacing the numerical weights of traditional multilayer perceptrons with function weights.

[0073] In the above-mentioned text-driven image style transfer method based on multi-level feature fusion and gating, the KAN is used to process the preliminary fused features of the text image. This process can perform nonlinear transformation and dimensional adjustment on the preliminary fused features, enhance the expressive power of the features, optimize the distribution and representation dimension of the features, and improve the expressive power and style detail transmission ability of the preliminary fused features, thereby making the style expression more delicate and accurate.

[0074] In an optional embodiment of this application, please refer to Figure 6 , Figure 7 , Figure 8 and Figure 9 The image features may include a first image feature, a second image feature, a third image feature, a fourth image feature, and a fifth image feature. The STAG may include a multi-scale fusion submodule, which may include a low-level multi-scale fusion submodule, a mid-level multi-scale fusion submodule, and a high-level multi-scale fusion submodule. The STAG may include a multi-scale fusion submodule, a feature enhancement submodule, and a linear mapping submodule. The image features are input into the texture semantic adaptive gating module to generate texture semantic fusion image features, which may include: Step S601: Input the first image feature, the second image feature, and the third image feature into the low-level multi-scale fusion submodule to generate low-level image fusion features, and input the third image feature, the fourth image feature, and the fifth image feature into the high-level multi-scale fusion submodule to generate high-level image fusion features.

[0075] Optionally, taking the VGG image encoder as an example of a neural network model constructed from the VGG19 model, the first image feature can be the feature output by the ReLU activation layer following the first convolutional layer of the first convolutional block of VGG19; the second image feature can be the feature output by the ReLU activation layer following the first convolutional layer of the second convolutional block of VGG19; the third image feature can be the feature output by the ReLU activation layer following the first convolutional layer of the third convolutional block of VGG19; the fourth image feature can be the feature output by the ReLU activation layer following the first convolutional layer of the fourth convolutional block of VGG19; and the fifth image feature can be the feature output by the ReLU activation layer following the th convolutional layer of the 5th convolutional block of VGG19.

[0076] Optionally, the expression for the multi-scale fusion submodule can be:

[0077]

[0078] In the formula, The output features of the multi-scale fusion submodule, As a multi-scale fusion submodule, , and These represent the input features with the lowest semantic level, the input features with the middle semantic level, and the input features with the highest semantic level in the multi-scale fusion sub-module, respectively. For the channel adaptive weights of the multi-scale fusion submodule, , is the activation function of the activation function layer of the multi-scale fusion submodule.

[0079] For example, with Figure 7 Taking the low-level multi-scale fusion submodule in the middle as an example, combined with Figure 8 To clarify: The first input to the low-level multi-scale fusion submodule can be a first image feature, the second input can be a second image feature, and the third input can be a third image feature. The low-level multi-scale fusion submodule can generate channel adaptive weights based on the second input through an activation function layer, and can then perform weighted fusion of the first and third inputs based on these generated channel adaptive weights.

[0080] Step S602: Input the third image feature, high-level image fusion feature and low-level image fusion feature into the mid-level multi-scale fusion submodule to obtain the image fusion feature.

[0081] For example, the first input of the mid-level multi-scale fusion submodule can be the low-level image fusion features generated by the low-level multi-scale fusion submodule, the second input of the mid-level multi-scale fusion submodule can be the third image features, and the third input of the mid-level multi-scale fusion submodule can be the high-level image fusion features generated by the high-level multi-scale fusion submodule. The mid-level multi-scale fusion submodule can generate channel adaptive weights for itself based on the second input through an activation function layer, and can perform weighted fusion of the first and third inputs based on the generated channel adaptive weights to obtain the image fusion features.

[0082] Step S603: Based on the feature enhancement submodule of the texture semantic adaptive gating module, the image fusion features are enhanced to generate fusion enhanced features.

[0083] Specifically, the image transfer terminal can input the image fusion features generated by the mid-level multi-scale fusion submodule into the STAG feature enhancement submodule (FE submodule) to enhance the image fusion features and generate fusion enhanced features.

[0084] Furthermore, such as Figure 9 As shown, the FE submodule can be based on a normalized layer and a convolutional kernel size of The K1 convolutional layer preprocesses the image fusion features, generating channel-adjusted base image fusion features. The FE submodule can be based on multi-channel convolutional kernels with a size of... The K3 depthwise convolutional layer, concatenation layer, and K1 convolutional layer perform multi-branch fusion enhancement on the channel-adjusted basic image fusion features, resulting in multi-branch fusion enhanced image fusion features. The FE submodule can also perform global enhancement on the multi-branch fusion enhanced image fusion features based on an attention mechanism constructed through global pooling layers and K3 depthwise convolutions, generating globally enhanced image fusion features. Finally, the FE submodule can obtain fusion enhanced features based on the residual connections between the image fusion features and the globally enhanced image fusion features.

[0085] Step S604: The linear mapping submodule based on the texture semantic adaptive gating module performs linear mapping on the fusion enhancement features to obtain texture semantic fusion image features.

[0086] In the aforementioned text-driven image style transfer method based on multi-level feature fusion and gating, image features at different semantic levels are fused hierarchically through high-level, low-level, and mid-level multi-scale fusion sub-modules. This enables precise complementarity of features at different levels, thereby generating image fusion features that contain both global structural information and local texture details. The STAG FE sub-module enhances the image fusion features, strengthening the representation strength of key information in the fusion features and weakening the interference of redundant information, thus generating more information-focused fusion enhancement features. The linear mapping sub-module performs linear mapping on the fusion enhancement features, adjusting the dimension and distribution of the features to adapt them to the input requirements of the subsequent ITHFN module, thereby generating texture semantic fusion image features that meet the subsequent fusion requirements.

[0087] In an optional embodiment of this application, please refer to Figure 3 and Figure 10 Text-driven image style transfer methods based on multi-level feature fusion and gating can also include: Step S308: Construct a text-guided style transfer framework based on a visual geometry group image encoder, a contrastive language image pre-trained text encoder, a texture semantic adaptive gating module, a multimodal feature coordinator, an image-text hierarchical fusion network module, and a decoder.

[0088] Optionally, the loss function of the text-guided style transfer framework may include a style contrast loss function and an image content loss function.

[0089] Step S309: Input the style image into the visual geometry group image encoder to obtain style image features, and train the text-guided style transfer framework based on the style image features and image features, combined with the image content loss function.

[0090] Specifically, the image transfer terminal can input the style image generated by the text-guided style transfer framework into the VGG image encoder to obtain style image features, and then train the text-guided style transfer framework based on the style image features and image features, combined with the image content loss function.

[0091] Step S310: Input the style image into the contrastive language image pre-trained image encoder to obtain the contrastive language image features, and train the text-guided style transfer framework based on the contrastive language image features and text features, combined with the style contrast loss function.

[0092] Specifically, the image transfer terminal can input the style image generated by the text-guided style transfer framework into the CLIP image encoder to obtain contrastive language image features. Based on the contrastive language image features and text features, and combined with the style contrast loss function, the text-guided style transfer framework is trained.

[0093] Optionally, the expressions for the image content loss function and the style contrast loss function can be:

[0094]

[0095]

[0096]

[0097]

[0098] In the formula, The image content loss function is... The number of feature extractor layers in the VGG image encoder used in the calculation. For the VGG image encoder Layer feature extractor, For content images, For style images. Let style contrast loss function be used. For temperature hyperparameters, This is the dot product operator. , and These are, respectively, the contrastive language image pre-trained features of style images, the contrastive language image pre-trained features of text descriptions, and the first... Contrastive language image pre-trained features of negative text samples. The total number of negative samples in the text. and These are CLIP image encoders and CLIP image encoders, respectively. and They are respectively text description and the first Negative samples of the text.

[0099] In the above text-driven image style transfer method based on multi-level feature fusion and gating, a loss function is used... and Training a text-guided style transfer framework enables precise bidirectional constraints on the results generated by the framework. First, the style image is input into a VGG image encoder to obtain style image features, which are then combined with… By comparing style image features with original image features, the model can be constrained to preserve the core structure of the content image during style transfer by calculating the difference between the two. Secondly, the style image is input into the CLIP image encoder to obtain style features (compared to language image features), and then combined with... Comparing the CLIP features of style images, text descriptions, and negative text samples with language image features can enhance the similarity between style image features and target text features, weaken their correlation with negative sample features, constrain the semantic consistency between style and text descriptions, and improve the semantic alignment of style transfer.

[0100] In an optional embodiment of this application, the loss function may further include a contrastive language image pre-training direction loss function, a contrastive language image pre-training local block loss function, and a difference loss function.

[0101] Optionally, the expressions for the contrastive language image pre-training orientation loss function, the contrastive language image pre-training local block loss function, and the difference loss function can be:

[0102]

[0103]

[0104]

[0105]

[0106]

[0107] In the formula, , and These are the contrastive language image pre-training direction loss function, the contrastive language image pre-training local block loss function, and the difference loss function, respectively. For text description and content images Style description benchmark text The direction of text style changes between them For style images and content images The direction of the change in image style between For content images The style description of the benchmark text, This represents the total number of local blocks. For style images The Block local block and content image The Differences in style orientation between local block images of different blocks. and Style images The Block local block and content image The Block local block, For random perspective enhancement function, The total number of difference directions. For the first Gradient operators in each direction.

[0108] Optional, gradient operator This can be a first-order difference between adjacent pixels in the top, bottom, left, and right directions of an image. In this case, the total number of difference directions is... It can be 4.

[0109] Schematic, random perspective enhancement function It is an image spatial geometry enhancement operator that can simulate the perspective distortion caused by three-dimensional perspective transformation by randomly moving the positions of the four corner points of the image on the two-dimensional projection plane. This allows it to generate samples with depth and viewpoint diversity without changing the semantic labels of the image. It is widely used in the training stage of tasks such as image classification, object detection, and style transfer.

[0110] In the above text-driven image style transfer method based on multi-level feature fusion and gating, through , and It can enhance the control over style transfer results from three dimensions: global style direction, local style consistency, and visual quality. Among these, By calculating changes in text style direction Changes in image style direction Cosine similarity can constrain the consistency between style images and text descriptions in the global style direction, ensuring that the overall direction of style transfer aligns with the style intent guided by the text. Figure 1 This ensures that the overall style of the generated image precisely matches the style guide of the text description; By analyzing local patches of the style image and content image, style direction differences are identified. Calculation can constrain the consistency between the style of local regions and the text description, which can avoid local style misalignment and thus improve the visual effect of style images. By calculating the gradient differences of style images in multiple directions and constraining their smoothness, artifacts, noise, or abrupt edge breaks in the generated images can be effectively suppressed. By controlling the gradient changes of adjacent pixels, the visual presentation of style images can be made more natural and smooth, thereby further improving the visual quality of style images.

[0111] In one exemplary embodiment of this application, such as Figure 3 As shown, a text-driven image style transfer method based on multi-level feature fusion and gating is provided, which may include: Step S301: Obtain the content image and text description. Input the content image into the visual geometry group image encoder to obtain image features, and input the text description into the contrastive language image pre-trained text encoder to obtain text features.

[0112] In step S302, the image features are input into the texture semantic adaptive gating module to generate texture semantic fusion image features, and the text features are input into the multimodal feature coordinator built based on the fully connected layer to generate modulation style text features.

[0113] Step S303: Input the modulation style text features and texture semantic fusion image features into the text semantic style injection submodule of the image text hierarchical fusion network module to generate text image fusion features.

[0114] Step S304: Input each text image fusion feature into the fusion feature enhancement submodule of the corresponding image-text hierarchical fusion network module to generate text image enhancement fusion features corresponding to each text image fusion feature.

[0115] Step S305: Input each text image enhancement fusion feature into the hierarchical feature interaction submodule of the image-text hierarchical fusion network module to obtain text image hierarchical fusion features.

[0116] Step S306: After concatenating the modulation style text features and text image hierarchical fusion features, perform feature fusion with the image features to generate image transfer features.

[0117] Step S307: Input the image transfer features into the decoder built on the convolutional neural network to generate a style image.

[0118] Step S308: Construct a text-guided style transfer framework based on a visual geometry group image encoder, a contrastive language image pre-trained text encoder, a texture semantic adaptive gating module, a multimodal feature coordinator, an image-text hierarchical fusion network module, and a decoder.

[0119] Step S309: Input the style image into the visual geometry group image encoder to obtain style image features, and train the text-guided style transfer framework based on the style image features and image features, combined with the image content loss function.

[0120] Step S310: Input the style image into the contrastive language image pre-trained image encoder to obtain the contrastive language image features, and train the text-guided style transfer framework based on the contrastive language image features and text features, combined with the style contrast loss function.

[0121] In the aforementioned text-driven image style transfer method based on multi-level feature fusion and gating, image and text features are extracted by the VGG image encoder and CLIP text encoder, respectively, and then processed by STAG and MMFH to achieve preliminary cross-modal feature adaptation, thus providing more adaptable basic features for subsequent cross-modal fusion. Through progressive processing including TeSSI, a fusion feature enhancement submodule, a hierarchical feature interaction module, and global fusion, multi-level precise fusion of text style and image content can be achieved, generating image transfer features with both rich detail and overall consistency. By processing the image transfer features using a decoder based on a multi-layer convolutional neural network, the abstract transfer features can be restored to a visualized style image, generating a visually coherent style image that conforms to the text style and retains the content structure. By combining... and Training the text-guided style transfer framework can efficiently constrain it, thereby improving the stability and accuracy of the style images generated by the framework.

[0122] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0123] Based on the same inventive concept, this application also provides a text-driven image style transfer system based on multi-level feature fusion and gating for implementing the text-driven image style transfer method based on multi-level feature fusion and gating described above. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the text-driven image style transfer system based on multi-level feature fusion and gating provided below can be found in the limitations of the text-driven image style transfer method based on multi-level feature fusion and gating described above, and will not be repeated here.

[0124] In one exemplary embodiment, such as Figure 11 As shown, a text-driven image style transfer system 110 based on multi-level feature fusion and gating is provided, comprising: The data feature extraction module 111 can be used to acquire content images and text descriptions, input the content images into the visual geometry group image encoder to obtain image features, and input the text descriptions into the contrastive language image pre-trained text encoder to obtain text features.

[0125] The data feature analysis module 112 can be used to input image features into the texture semantic adaptive gating module to generate texture semantic fusion image features, and input text features into the multimodal feature coordinator built based on fully connected layers to generate modulation style text features.

[0126] The data feature fusion module 113 can be used to input texture semantic fusion image features, modulation style text features and image features into the image text hierarchical fusion network module to obtain image transfer features.

[0127] The transfer result generation module 114 can be used to input image transfer features into a decoder built on a convolutional neural network to generate a style image, which is used to characterize the image style transfer result of the content image under the guidance of text description.

[0128] In an optional embodiment of this application, the data feature fusion module 113 may also be used for: Modulation style text features and texture semantic fusion image features are input into the text semantic style injection submodule of the image-text hierarchical fusion network module to generate text-image fusion features. Each text-image fusion feature is then input into the corresponding fusion feature enhancement submodule of the image-text hierarchical fusion network module to generate text-image enhancement fusion features corresponding to each text-image fusion feature.

[0129] Each text-image enhancement and fusion feature is input into the hierarchical feature interaction submodule of the image-text hierarchical fusion network module to obtain text-image hierarchical fusion features.

[0130] After feature concatenation of modulation style text features and text-image hierarchical fusion features, feature fusion is performed with image features to generate image transfer features.

[0131] In an optional embodiment of this application, the data feature fusion module 113 may also be used for: The texture semantic fusion image features are input into the multi-head self-attention layers of the fine text semantic style injection submodule, the Chinese text semantic style injection submodule, and the coarse text semantic style injection submodule. After passing through the residual connection layer and the normalization layer, the texture semantic fusion image attention features are obtained. The modulation style text features and texture semantic fusion image attention features are input into the multi-head cross attention layer of the coarse text semantic style injection submodule. After passing through the residual connection layer and the normalization layer, coarse-grained text image preliminary fusion features are generated. The coarse-grained text image preliminary fusion features are then input into the multilayer perceptron of the coarse text semantic style injection submodule to obtain coarse-grained text image fusion features. The coarse-grained text image fusion features and texture semantic fusion image attention features are input into the multi-head cross-attention layer of the Chinese text semantic style injection submodule. After passing through the residual connection layer and the normalization layer, the preliminary fusion features of the medium-grained text image are generated. The preliminary fusion features of the medium-grained text image are then input into the multilayer perceptron of the Chinese text semantic style injection submodule to obtain the medium-grained text image fusion features. The medium-grained text image fusion features and texture semantic fusion image attention features are input into the multi-head cross-attention layer of the fine text semantic style injection submodule. After passing through the residual connection layer and the normalization layer, the fine-grained text image preliminary fusion features are generated. The fine-grained text image preliminary fusion features are then input into the multilayer perceptron of the fine text semantic style injection submodule to obtain the fine-grained text image fusion features.

[0132] In an optional embodiment of this application, the data feature analysis module 112 may also be used for: The first, second, and third image features are input into the low-level multi-scale fusion submodule to generate low-level image fusion features, and the third, fourth, and fifth image features are input into the high-level multi-scale fusion submodule to generate high-level image fusion features.

[0133] The third image feature, high-level image fusion feature, and low-level image fusion feature are input into the mid-level multi-scale fusion submodule to obtain the image fusion feature.

[0134] The feature enhancement submodule based on the texture semantic adaptive gating module enhances the image fusion features and generates fusion enhanced features.

[0135] The linear mapping submodule based on the texture semantic adaptive gating module performs linear mapping on the fusion enhancement features to obtain texture semantic fusion image features.

[0136] In an optional embodiment of this application, the text-driven image style transfer system 110 based on multi-level feature fusion and gating can also be used for: A text-guided style transfer framework is constructed based on a visual geometry group image encoder, a contrastive language image pre-trained text encoder, a texture semantic adaptive gating module, a multimodal feature coordinator, an image-text hierarchical fusion network module, and a decoder. The loss function of the text-guided style transfer framework includes a style contrast loss function and an image content loss function.

[0137] The style image is input into the visual geometry group image encoder to obtain style image features. Based on the style image features and image features, combined with the image content loss function, a text-guided style transfer framework is trained.

[0138] The style image is input into the contrastive language image pre-trained image encoder to obtain the contrastive language image features. Based on the contrastive language image features and text features, combined with the style contrast loss function, the text-guided style transfer framework is trained.

[0139] In an optional embodiment of this application, the loss function further includes a contrastive language image pre-training direction loss function, a contrastive language image pre-training local block loss function, and a difference loss function.

[0140] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the text-driven image style transfer method based on multi-level feature fusion and gating as described above.

[0141] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0142] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0143] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A text-driven image style transfer method based on multi-level feature fusion and gating, characterized in that, The method includes: The content image and text description are acquired. The content image is input into the visual geometry group image encoder to obtain image features, and the text description is input into the contrastive language image pre-trained text encoder to obtain text features. The image features are input into the texture semantic adaptive gating module to generate texture semantic fusion image features, and the text features are input into the multimodal feature coordinator built based on a fully connected layer to generate modulation style text features; The texture semantic fusion image features, the modulation style text features, and the image features are input into the image-text hierarchical fusion network module to obtain image transfer features; The image transfer features are input into a decoder built on a convolutional neural network to generate a style image, which is used to characterize the image style transfer result of the content image under the guidance of the text description.

2. The method according to claim 1, characterized in that, The step of inputting the texture semantic fusion image features and the modulation style text features into the image-text hierarchical fusion network module to obtain image transfer features includes: The modulation style text features and the texture semantic fusion image features are input into the text semantic style injection submodule of the image-text hierarchical fusion network module to generate text-image fusion features; wherein, the text semantic style injection submodule includes a fine text semantic style injection submodule, a medium text semantic style injection submodule and a coarse text semantic style injection submodule, and the text-image fusion features include fine-grained text-image fusion features, medium-grained text-image fusion features and coarse-grained text-image fusion features; Each of the text image fusion features is input into the fusion feature enhancement submodule of the corresponding image text hierarchical fusion network module to generate text image enhancement fusion features corresponding to each of the text image fusion features; Each of the text image enhancement and fusion features is input into the hierarchical feature interaction submodule of the image-text hierarchical fusion network module to obtain text image hierarchical fusion features. The text image hierarchical fusion features are used to characterize the multi-level adaptive fusion result of the fine-grained text image fusion features, the medium-grained text image fusion features, and the coarse-grained text image fusion features. After concatenating the modulation style text features and the text image hierarchical fusion features, the text features are fused with the image features to generate the image transfer features. The expression for the fusion feature enhancement submodule is as follows: In the formula, For particle size number The text image enhancement fusion feature, For particle size number The text-image fusion features, For particle size number The multi-head self-attention layer of the aforementioned fusion feature enhancement submodule, For particle size number The layer normalization layer of the fusion feature enhancement submodule.

3. The method according to claim 2, characterized in that, The step of inputting the modulation style text features and the texture semantic fusion image features into the text semantic style injection submodule of the image-text hierarchical fusion network module to generate text-image fusion features includes: The texture semantic fusion image features are input into the multi-head self-attention layers of the fine text semantic style injection submodule, the Chinese text semantic style injection submodule, and the coarse text semantic style injection submodule, and after passing through the residual connection layer and the normalization layer, the texture semantic fusion image attention features are obtained. The modulation style text features and the texture semantic fusion image attention features are input into the multi-head cross-attention layer of the coarse text semantic style injection submodule. After passing through the residual connection layer and the normalization layer, coarse-grained text image preliminary fusion features are generated. The coarse-grained text image preliminary fusion features are then input into the multilayer perceptron of the coarse text semantic style injection submodule to obtain the coarse-grained text image fusion features. The coarse-grained text image fusion features and the texture semantic fusion image attention features are input into the multi-head cross-attention layer of the Chinese text semantic style injection submodule. After passing through the residual connection layer and the normalization layer, the preliminary fusion features of the medium-grained text image are generated. The preliminary fusion features of the medium-grained text image are then input into the multilayer perceptron of the Chinese text semantic style injection submodule to obtain the medium-grained text image fusion features. The medium-grained text image fusion features and the texture semantic fusion image attention features are input into the multi-head cross-attention layer of the fine text semantic style injection submodule. After passing through the residual connection layer and the normalization layer, the fine-grained text image preliminary fusion features are generated. The fine-grained text image preliminary fusion features are then input into the multilayer perceptron of the fine text semantic style injection submodule to obtain the fine-grained text image fusion features. The expression for the multi-head cross-attention layer is as follows: In the formula, For the multi-head cross-attention layer, , and These are, respectively, the query vector based on the texture semantic fusion image attention features, the key vector based on the modulation style text features, and the value vector based on the modulation style text features. For activation function, This is the transpose of the matrix. Let be the dimension of the multi-head cross-attention layer.

4. The method according to claim 3, characterized in that, The multilayer perceptron in the text semantic style injection submodule is replaced with a Kolmogorov-Arnold network.

5. The method according to claim 1, characterized in that, The image features include a first image feature, a second image feature, a third image feature, a fourth image feature, and a fifth image feature. The texture semantic adaptive gating module includes a multi-scale fusion sub-module, which further includes a low-level multi-scale fusion sub-module, a mid-level multi-scale fusion sub-module, and a high-level multi-scale fusion sub-module. The step of inputting the image features into the texture semantic adaptive gating module to generate texture semantic fused image features includes: The first image feature, the second image feature, and the third image feature are input into the low-level multi-scale fusion submodule to generate low-level image fusion features, and the third image feature, the fourth image feature, and the fifth image feature are input into the high-level multi-scale fusion submodule to generate high-level image fusion features. The third image feature, the high-level image fusion feature, and the low-level image fusion feature are input into the mid-level multi-scale fusion submodule to obtain the image fusion feature; Based on the feature enhancement submodule of the texture semantic adaptive gating module, feature enhancement is performed on the image fusion features to generate fusion enhanced features; The linear mapping submodule of the texture semantic adaptive gating module performs linear mapping on the fusion enhancement features to obtain the texture semantic fusion image features. The expression for the multi-scale fusion submodule is as follows: In the formula, The output features of the multi-scale fusion submodule are... For the multi-scale fusion submodule, , and These are, respectively, the input feature with the lowest semantic level among the input features of the multi-scale fusion submodule, the input feature with the middle semantic level among the input features of the multi-scale fusion submodule, and the input feature with the highest semantic level among the input features of the multi-scale fusion submodule. The channel adaptive weights of the multi-scale fusion submodule are... This is the activation function.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: A text-guided style transfer framework is constructed based on the visual geometry group image encoder, the contrastive language image pre-trained text encoder, the texture semantic adaptive gating module, the multimodal feature coordinator, the image-text hierarchical fusion network module, and the decoder. The loss function of the text-guided style transfer framework includes a style contrast loss function and an image content loss function. The style image is input into the visual geometry group image encoder to obtain style image features. Based on the style image features and the image features, combined with the image content loss function, the text-guided style transfer framework is trained. The style image is input into a contrastive language image pre-trained image encoder to obtain contrastive language image features. Based on the contrastive language image features and the text features, combined with the style contrast loss function, the text-guided style transfer framework is trained. The expressions for the image content loss function and the style contrast loss function are as follows: In the formula, Let be the image content loss function. The total number of feature extractors in the visual geometry group image encoder that participate in the computation. For the visual geometry group image encoder, the first Layer feature extractor, The content image, For the style image, Let the style contrast loss function be... For temperature hyperparameters, This is the dot product operator. , and These are respectively the contrastive language image pre-trained features of the style image, the contrastive language image pre-trained features of the text description, and the first... The contrastive language image pre-trained features of the negative samples of the text, The total number of negative samples in the text. and These refer to the contrastive language image pre-trained image encoder and the contrastive language image pre-trained image encoder, respectively. and The text description and the first Negative samples of the text.

7. The method according to claim 6, characterized in that: The loss function also includes a contrastive language image pre-training direction loss function, a contrastive language image pre-training local block loss function, and a difference loss function; The expressions for the contrastive language image pre-training direction loss function, the contrastive language image pre-training local block loss function, and the difference loss function are as follows: In the formula, , and These are the contrastive language image pre-training direction loss function, the contrastive language image pre-training local block loss function, and the difference loss function, respectively. The text style variation between the reference text and the content image is defined as the directional change in text style. This refers to the directional change in image style between the style image and the content image. The style description reference text for the content image. This represents the total number of local blocks. For the style image of the first The block local block and the first of the content images Differences in style orientation between local block images of different blocks. and The first style image is respectively The block local block and the first of the content images Block local block, For random perspective enhancement function, The total number of difference directions. For the first Gradient operators in each direction.

8. A text-driven image style transfer system based on multi-level feature fusion and gating, characterized in that, The system includes: The data feature extraction module is used to acquire content images and text descriptions, input the content images into the visual geometry group image encoder to obtain image features, and input the text descriptions into the contrastive language image pre-trained text encoder to obtain text features; The data feature analysis module is used to input the image features into the texture semantic adaptive gating module to generate texture semantic fusion image features, and to input the text features into a multimodal feature coordinator built based on a fully connected layer to generate modulation style text features; The data feature fusion module is used to input the texture semantic fusion image features, the modulation style text features, and the image features into the image text hierarchical fusion network module to obtain image transfer features; The migration result generation module is used to input the image migration features into a decoder built based on a convolutional neural network to generate a style image, which is used to characterize the image style migration result of the content image under the guidance of the text description.

Images