A method of embedding a reconfigured text-image alignment style transfer

Abstract

An innovative embedded reconstruction text-image alignment style transfer method, comprising: constructing a basic framework based on a diffusion model, adding noise and denoising through a forward process to learn image style features. Use a perceptron attention and a feedforward network to extract style embeddings from an art dataset. Generate text embeddings through a text encoder to provide guidance for style transfer. Use a cross-attention mechanism to fuse style embeddings with text embeddings to generate multi-modal embeddings, optimizing the interaction of text and image information. Linearly interpolate multi-modal embeddings and image embeddings, and splice them with text embeddings to form complete prompt embeddings, which are integrated into the diffusion model to generate style images that match the text description. The method significantly improves the quality and diversity of artistic image style transfer, has wide application potential in the field of artistic creation and design, and can provide a wide range of stylized visual results to meet the needs of diverse creative scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence computer vision processing, in particular to a method for embedding and reconstructing text-image alignment style transfer. BACKGROUND

[0002] In the field of text-guided image style transfer, some existing methods use a traditional pre-trained diffusion model combined with a simple adapter module to handle the style transfer task. However, there are significant drawbacks: on the one hand, the model architecture and training data are adapted to the natural image characteristics, and when facing artistic images, it is difficult to capture unique style details such as oil painting strokes and Chinese painting atmosphere, and abstract style concepts, resulting in a lack of artistic appeal and low style restoration of the style transfer result. On the other hand, when processing image and text embedding, the information imbalance between the two is not effectively compensated, and direct splicing causes the text guide to be easily weakened by image data in the generation process, making it impossible to accurately drive the style and content to be shaped according to the text semantic, resulting in a single image style and poor fit with the text description, greatly limiting the deep expansion and efficient application of style transfer technology in artistic creation, design creativity and other diverse scenarios.

[0003] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0004] The main purpose of the present application is to overcome the defects in the background art, and to provide a method for embedding and reconstructing text-image alignment style transfer.

[0005] To achieve the above purpose, the present application adopts the following technical solutions:

[0006] A method for embedding and reconstructing text-image alignment style transfer, comprising the following steps:

[0007] S1. Building a basic model: using a diffusion model as the basic architecture, the model gradually adds noise to the image data through the forward process, simulates the gradual transition of the image from a clear state to a completely noisy state, and then restores the original image from the noisy image through the denoising process, achieving learning and coding of image style features;

[0008] S2. Style extraction based on attention: for the reference image in the artwork dataset, extract the input image embedding through the feature extraction network, which is used to capture the visual features of the image; then use the perceptual attention mechanism and the feedforward network FFN to further process the image embedding, capturing complex style details from it, and generating a style embedding;

[0009] S3. Text embedding generation: The text description is converted into text embeddings through a text encoder to guide the style transfer process;

[0010] S4. Text-image alignment enhancement: The style embedding is fused with the text embedding through a cross-attention mechanism to generate a multimodal embedding to optimize the interaction between text and image information.

[0011] S5. Explicit modulation and diffusion model integration: The multimodal embedding is fused with the image embedding via linear interpolation to form a complete hint embedding, which is then integrated into the diffusion model to generate diverse style images that match the text description.

[0012] Furthermore, the diffusion model uses U-Net to implement the denoising process and is trained through mean square error loss.

[0013] Furthermore, step S2 specifically includes:

[0014] Generate an initial latent variable for the reference image, which is normalized to stabilize the training process;

[0015] Expand the latent variables to match the batch size of the input images to maintain consistency when processing multiple images;

[0016] Updating the latent variables through the perceptron attention mechanism enables the model to selectively focus on different parts of the input image based on the learnable latent variables;

[0017] The updated latent variables are transformed nonlinearly through the feedforward network FFN to enhance the style encoding capability;

[0018] The output of the feedforward network and the updated latent variables are combined to generate an embedding representing the style of the input image for subsequent text-image alignment enhancement.

[0019] Furthermore, in step S2, the visual part of the CLIP model is used as a feature extraction network to obtain an embedded representation of the input image as a reference basis for style extraction; the output of the perceptron attention mechanism is further processed through a position-by-position feedforward network FFN to capture more fine-grained style features; the output of the FFN is combined with the updated latent variables to obtain the final style embedding.

[0020] Furthermore, the feedforward network FFN includes at least two linear transformation steps for weighting the latent variables, and introduces a nonlinear activation function between the two linear transformations to enhance the model's ability to learn complex features.

[0021] Furthermore, step S4 specifically includes:

[0022] The style embedding and the text embedding are converted into query, key and value matrices respectively through linear layers to interact in a shared feature space;

[0023] The attention weights are calculated using the dot product of the query matrix and the key matrix, and scaling is used to prevent gradient vanishing, thus dynamically prioritizing different aspects of the text prompt;

[0024] A softmax function is applied to the original attention scores to obtain normalized attention weights, ensuring that the sum of the weights is 1 for effective information integration;

[0025] The value matrix is weighted summed using the normalized attention weights to generate a multimodal embedding that can more effectively capture the multimodal context of text and image.

[0026] Through the above process, the model can more effectively integrate image style and text embedding, allowing the generation of images that are more consistent with the semantic content of the text prompt, especially suitable for scenarios that require close integration of text and visual information.

[0027] Further, step S5 specifically includes

[0028] The style embedding is fused with the multimodal embedding obtained by the text-image alignment enhancement through linear interpolation to achieve smooth transition and effective combination between the two embeddings;

[0029] A predefined constant is used to control the fusion ratio between the style embedding and the multimodal embedding, ensuring that the flexibility of fusion can be adjusted according to different application scenarios and requirements;

[0030] The fused image embedding is spliced with the text prompt embedding to form a complete prompt embedding for image generation, which integrates the multimodal information of text and image;

[0031] The generated complete prompt embedding is integrated into the diffusion model, allowing the model to consider the multimodal conditions of text and image during the generation process, improving the performance and quality of the generated image.

[0032] Through the above fusion and splicing operations, the model can robustly and controllably capture the multimodal conditions, thus achieving effective alignment of text description and image style in the image generation process.

[0033] A computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the embedding reconstruction text-image alignment style transfer method.

[0034] A computer program product comprising a computer program, the computer program being executed by a processor to implement the embedding reconstruction text-image alignment style transfer method.

[0035] The present application has the following advantages:

[0036] The present application provides a method of embedding reconstructed text-image alignment style transfer, which innovatively solves the problem of inaccuracy and incompleteness of existing technology in artistic image style feature extraction. By constructing a style extraction module based on a perceptual attention and multi-layer architecture, the capture ability of complex artistic image style details and abstract characteristics is significantly improved, thereby achieving high-fidelity and diversified presentation of artistic style in style transfer. Through the design of a sophisticated text-image alignment enhancement module, the present application optimizes the fusion process of text and image embedding using a cross-attention mechanism, strengthens the guiding role of text throughout the image generation process, and ensures that the generated image closely matches the text semantic guidance, significantly improving the controllability and accuracy of style transfer. The introduction of an explicit modulation module realizes the flexible fusion of multi-modal embedding and original embedding through a linear interpolation fusion rule and a fusion ratio control mechanism, balancing content and style so that the generated image maintains the core content while exhibiting rich style variations. These technical advantages not only enrich the style diversity of creative materials, but also improve the accuracy of the combination of creative content and creative ideas, expanding the deep development and efficient application of style transfer technology in artistic creation, design creativity and other diverse scenarios, thereby improving the practical value and innovative potential of style transfer technology from multiple dimensions.

[0037] The significant advantages of the present application include the following aspects:

[0038] (1) Excellent style representation

[0039] The style extraction module based on a unique architecture far surpasses traditional technology in artistic style coding, accurately capturing complex artistic image style details and abstract characteristics, resulting in vivid artistic style of generated images and delicate and realistic style transfer effects, greatly enriching the style diversity of creative materials.

[0040] (2) Accurate and efficient text guidance

[0041] The text-image alignment enhancement module successfully reverses the text guidance disadvantage, strengthens the control of text at each stage of image generation, ensures that the generated image faithfully echoes the text description, improves the accuracy of the combination of creative content and creative ideas, and expands the depth of application of style transfer in the field of creative design.

[0042] (3) Output is rich and flexible

[0043] The dynamic fusion mechanism of the explicit modulation module unlocks various style presentations, maintains the content core while giving the generated image rich style variants, provides users with a large number of stylized visual results, and effectively supports the style innovation needs of multi-creation scenarios, and improves the practical value and innovation potential of the style transfer technology from multiple dimensions.

[0044] Other benefits of the embodiments of the present application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The text-image style transfer method based on the attention mechanism of the embodiments of the present application is a general flowchart.

[0046] Figure 2 The algorithm architecture diagram of the text-image alignment enhancement and explicit modulation of the embodiments of the present application.

[0047] Figure 3 The visual results of the embodiments of the present application.

[0048] Figure 4 The embodiments of the present application and the comparison of the results generated based on the general adapter.

[0049] Figure 5 The qualitative comparison of the embodiments of the present application and other most advanced text-guided stylization methods.

[0050] Figure 6 The visual difference between the samples with global features and attention-based style features generated by the embodiments of the present application.

[0051] Figure 7 The ablation study visualization diagram of the text-image alignment enhancement module of the embodiments of the present application.

[0052] Figure 8 The ablation study heat map of the text-image alignment enhancement module of the embodiments of the present application.

[0053] Figure 9 The effect diagram of different degrees of explicit modulation in the embodiments of the present application.

[0054] Figure 10 The visual results of the embodiments of the present application trained under different data sets respectively.

[0055] Figure 11 The guided results of the embodiments of the present application combined with additional conditions. DETAILED DESCRIPTION

[0056] The embodiments of the present application will be described in detail below. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present application and its applications.

[0057] In addition, the terms "first", "second", etc. are used only for descriptive purposes and should not be construed as indicating or implying relative importance or an indicated number of technical features. Therefore, the features defined as "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0058] Referring to Figure 1 The embodiments of the present application provide a method for embedding and reconstructing text-image alignment style transfer, comprising the following steps:

[0059] Step S1. Building a basic model: a diffusion model is used as the basic architecture, which gradually adds noise to image data through a forward process, simulates the gradual transition of images from a clear state to a completely noisy state, and then recovers the original image from the noisy image through a denoising process, achieving learning and encoding of image style features.

[0060] In a preferred embodiment, the diffusion model uses U-Net to implement the denoising process and is trained through mean square error loss.

[0061] Step S2. Style extraction based on attention: for the reference image in the art data set, the input image embedding is extracted through a feature extraction network to capture the visual features of the image; then the image embedding is further processed by using a perceptual attention mechanism and a feedforward network FFN to capture complex style details, generating a style embedding.

[0062] In a preferred embodiment, step S2 specifically comprises: generating an initial latent variable for the reference image, which is normalized to stabilize the training process; expanding the latent variable to match the batch size of the input images to maintain consistency when processing multiple images; updating the latent variable through a perceptual attention mechanism, enabling the model to selectively focus on different parts of the input image based on the learnable latent variable; performing a non-linear transformation on the updated latent variable through a feed-forward network FFN, enhancing the style encoding capability; combining the output of the feed-forward network and the updated latent variable to generate an embedding representing the style of the input image, which is used for subsequent text-image alignment enhancement. In a further preferred embodiment, the visual part of the CLIP model is used as a feature extraction network to obtain the embedding representation of the input image as the reference basis for style extraction; the output of the perceptual attention mechanism is further processed through a position-wise feed-forward network FFN to capture more fine-grained style features; the output of the FFN is combined with the updated latent variable to obtain the final style embedding. Preferably, the feed-forward network FFN comprises at least two linear transformation steps for weighted processing of the latent variable, and a non-linear activation function is introduced between the two linear transformations to enhance the model's learning ability for complex features.

[0063] Step S3. Text embedding generation: converting the text description into a text embedding through a text encoder, which is used to guide the style transfer process.

[0064] Step S4. Text-image alignment enhancement: fusing the style embedding obtained in step S2 and the text embedding in step S3 through a cross-attention mechanism to generate a multi-modal embedding, optimizing the interaction between text and image information.

[0065] In a preferred embodiment, step S4 specifically comprises: converting the style embedding (image prompt embedding) and the text embedding into query, key and value matrices respectively through a linear layer, so as to interact in a shared feature space; calculating the attention weights using the dot product of the query matrix and the key matrix, and preventing gradient vanishing through scaling, so as to dynamically prioritize different aspects of the text prompt; applying a softmax function to the original attention scores to obtain normalized attention weights, ensuring that the sum of the weights is 1, so as to effectively integrate information; using the normalized attention weights to perform weighted summation on the value matrix to generate a multi-modal embedding, which can more effectively capture the multi-modal context of text and image. Through the above process, the model can more effectively integrate image and text embeddings, allowing the generation of images that are more consistent with the semantic content of the text prompt, which is particularly suitable for scenarios that require close integration of text and visual information.

[0066] Step S5. Explicit modulation and diffusion model integration: the multimodal embedding in step S4 is fused with the style embedding by linear interpolation, the image embedding obtained after fusion is spliced with the text prompt embedding to form a complete prompt embedding, and then the prompt embedding is integrated into the diffusion model to generate a diversified style image matching the text description.

[0067] In a preferred embodiment, step S5 specifically includes: fusing the style embedding with the text-image alignment enhanced multimodal embedding by linear interpolation to achieve smooth transition and effective combination between the two embeddings; using a predefined constant to control the fusion ratio between the style embedding and the multimodal embedding, ensuring that the flexibility of fusion can be adjusted according to different application scenarios and needs; splicing the image embedding obtained after fusion with the text prompt embedding to form a complete prompt embedding for image generation, which integrates the multimodal information of text and image; integrating the generated complete prompt embedding into the diffusion model, so that the model can consider the multimodal conditions of text and image during generation, improving the performance and quality of the generated image. Through the above fusion and splicing operations, the model can robustly and controllably capture the multimodal conditions, thereby achieving effective alignment of text description and image style during image generation.

[0068] The method of embedding reconstruction text-image alignment style transfer of the present application overcomes the problem of inaccurate and incomplete extraction of artistic image style features in the prior art, constructs an effective mechanism that can deeply mine and accurately encode artistic image style elements, and realizes high-fidelity and diversified presentation of artistic style in style transfer. The present application resolves the information imbalance dilemma when fusing text and image embeddings, designs a sophisticated text-image alignment strategy, strengthens the core regulatory position of text guidance throughout the image generation process, ensures that the generated image closely follows the text semantic guidance, and significantly improves the controllability and accuracy of style transfer.

[0069] The following further describes algorithm examples and experimental verification of specific embodiments of the present application.

[0070] The present application is built on the diffusion model infrastructure and mainly includes three core modules: a style extraction module based on attention, which captures multi-level style elements of images through perceptual attention and multi-layer architecture; a text-image alignment enhancement module, which optimizes the fusion and interaction of text and image embeddings using cross-attention mechanism; an explicit modulation module that fuses multimodal embeddings to improve image quality and diversity using linear interpolation and splicing strategy.

[0071] Base model

[0072] The diffusion model consists of two processes: a forward process that gradually adds Gaussian noise ∈ to the data x0 through Markov chains. In addition, a denoising process that removes Gaussian noise xT ~N(0, 1) samples, using a learnable denoising model ∈ θ (x t , t, c), parameterized by θ. This denoising model ∈ θ (·) is implemented with a U-Net and trained with a mean squared error loss derived from a simplified variational bound:

[0073]

[0074] where c denotes an optional conditioning. In the diffusion model, c is typically represented by a text embedding E t encoded using CLIP from a text prompt and integrated into the diffusion model through the following design module.

[0075] Attention-based style extraction

[0076] The style extraction method enhances the style encoding capacity by integrating fine-grained features from multiple layers. The goal is to capture complex style details from images using perceptual attention and position-wise feed-forward networks (FFN).

[0077] Given a reference image, the input image embedding is obtained through CLIP, denoted as x. The latent variable z is initialized as a tensor of shape (1, N, D), where N is the number of queries and D is the dimension of the latent space, and is normalized by dividing by the square root of D to stabilize the training process:

[0078]

[0079] To match the batch size of the input x, the latent variable z is expanded by repeating it in the batch dimension. This process can be represented as:

[0080]

[0081] where 1 B is a tensor of shape (B, 1, 1) of all 1s, and B is the batch size of x. The result of this operation is a tensor z of shape (B, N, D), where N is the number of queries and D is the dimension of the latent space.

[0082] Then apply the perceptual attention mechanism denoted as P-Attn to update the latent variable by focusing on the input x and the repeated latent variable:

[0083]

[0084] z′=P-Attn(x, z) + z,

[0085] where d kis the dimension of the key tensor, usually equal to D. This operation allows the model to selectively focus on different parts of the input data depending on the learnable latent variables.

[0086] The FFN consists of two linear transformations with a GELU activation function in between:

[0087] FFN(z′) = W2·GELU(W1·z′+b1) + b2,

[0088] where W1 and W2 are weight matrices and b1 and b2 are bias terms.

[0089] Output E I represents the style embedding extracted from the input image, obtained by combining the FFN output and the updated latent variable z′:

[0090] E I = FFN(z′) + z′.

[0091] Text-image alignment augmentation

[0092] The text-image alignment augmentation method aims to dynamically prioritize different aspects of the text prompt by leveraging a cross-attention mechanism. This module allows the model to more effectively integrate the image and text embeddings, projecting them into a shared feature space so that their interaction can be more nuanced.

[0093] The image prompt embedding E I and the text prompt embedding E T are first transformed into query, key, and value matrices by linear layers. These transformations are represented as:

[0094]

[0095] and are weight matrices associated with the image query, text key, and text value, respectively.

[0096] The attention weights w I are computed by taking the dot product of the query matrix Q I and the key matrix and scaled by the square root of the key dimension to prevent gradient vanishing:

[0097]

[0098] A softmax function is then applied to these raw attention scores to obtain normalized attention weights w′ I :

[0099] w′ I = softmax(wI )·

[0100] The weighted sum of the value matrices V I is computed using the normalized attention weights w T to generate a multimodal embedding E′ IT :

[0101] E′ IT = w′ I · V T·

[0102] E′ IT more effectively captures the multimodal context, allowing the model to generate images that are more consistent with the semantic content of the text prompt. This approach is particularly beneficial in scenarios where tight integration of text and visual information is required to produce coherent outputs.

[0103] Explicit modulation

[0104] Explicit modulation addresses the lack of flexibility in traditional fusion methods by seamlessly blending the image prompt embedding with the text-image alignment enhanced multimodal embedding using linear interpolation.

[0105] Specifically, the image prompt embedding E I is fused with the multimodal embedding E′ IT through linear interpolation:

[0106] E F = αE I + (1 - α)E′ IT ,

[0107] where α is a predefined constant controlling the fusion ratio between the original embedding and the enhanced embedding.

[0108] Finally, the fused image embedding E F is concatenated with the text prompt embedding E T to form the complete prompt embedding for image generation:

[0109]

[0110] where denotes the concatenation operation, E P represents the enhanced embedding, and is integrated into the diffusion model. By balancing the above embeddings, the model obtains a robust and controllable representation that effectively captures the multimodal conditions and improves the generation performance.

[0111] Other embodiments

[0112] Attention mechanism optimization: In the style extraction module, fine-tune the number of heads, key-value dimensions, and other hyperparameters of the perceptual attention, or introduce multi-head attention variants to balance the computational cost and style capture accuracy. The alignment enhancement module attempts to improve position encoding or hybrid attention mode to enhance the ability to process long text and complex images, and moderately increases the model complexity to improve the style transfer effect.

[0113] Fusion strategy improvement: The explicit modulation module explores nonlinear interpolation functions or adaptive fusion weight calculation methods to dynamically adapt the fusion strength based on image content and text semantics, or combines image semantic segmentation results to guide embedded fusion region selection, to improve the quality of generated images with more intelligent fusion strategies, although the design complexity is increased, but the technical optimization direction is expanded.

[0114] Data augmentation

[0115] Data set expansion and screening: Expand the artistic data set, include rare artistic styles, specific cultural images, and multi-modal derivative data, and optimize the data distribution through intelligent screening and weighted sampling to improve the model's adaptability to multiple styles. At the same time, use image enhancement techniques to enrich the texture, color, perspective, and other changes of training samples, enhance the model's generalization and robustness, and strengthen the model's style learning ability under the premise of increasing data processing.

[0116] Multi-modal data fusion optimization: Explore shallow fusion of image and text multi-modal features in the data preprocessing stage, or introduce external knowledge graphs, semantic networks, and other multi-modal knowledge in specific stages of model training to assist the model in building better text-image semantic mapping. Innovate the timing and method of multi-source data fusion to improve the depth and breadth of style transfer knowledge and expand the technical boundaries.

[0117] Experimental results

[0118] Table 1

[0119] CLIP-Text↑ CLIP-Image↑ Dinov2↑ LPIPS↑ The present invention 22.57 69.48 40.92 0.4908 StyleShot 22.46 59.47 23.45 0.4561 Style Aligned 18.26 63.82 27.89 0.3892 VSP 22.29 66.31 35.34 0.2653 InstantStyle 19.78 62.59 29.76 0.1478 InstantStyle(Plus) 19.53 64.65 30.62 0.1234 IP Adapter 15.01 68.90 38.12 0.3451 IP-Adapter(SDXL) 19.14 67.43 32.98 0.2987 CSGO 17.82 55.16 24.53 0.3655 StyleCrafter 19.37 58.68 31.6 0.3194

[0120] Table 1 shows the quantitative comparison of the performance of text and image style transfer between the present invention and the prior art, and the present invention exhibits superior performance in various indicators.

[0121] In summary, the present invention proposes a method of embedding and reconstructing text-image alignment style transfer, which has the following important innovative contributions compared to traditional technologies:

[0122] The style extraction module design and implementation process based on perceptual attention and multi-layer architecture, including latent variable processing, attention calculation, and feedforward network fusion.

[0123] The cross-attention mechanism architecture, matrix transformation process and weight calculation optimization strategy of the text-image alignment enhancement module ensure efficient interaction and fusion of text-image embedding in a shared feature space.

[0124] The linear interpolation fusion rule, fusion ratio regulation mechanism and embedding splicing method with the diffusion model of the explicit modulation module realize flexible fusion of multi-modal embedding and original embedding to balance content and style.

[0125] The application scenarios of the present application include but are not limited to:

[0126] Art creation assistance: Provide various style templates and intelligent style conversion tools for digital painting and illustration design to stimulate artists' creativity and improve the efficiency and diversity of creation.

[0127] Film and television special effect production: applied in the stylized processing of film and television scene and character special effects, accurately migrates styles according to the needs of the plot and character shaping, and enhances visual narration and artistic appeal.

[0128] Virtual fashion design: helps fashion designers quickly build virtual clothing series, integrates multiple fashion styles and popular elements, realizes design concept visualization and diversification, and innovates fashion creation process.

[0129] The embodiment of the present application also provides a storage medium for storing a computer program, which is executed to perform at least the method described above.

[0130] The embodiment of the present application also provides a control device, which includes a processor and a storage medium for storing a computer program; wherein the processor is used to execute the computer program to perform at least the method described above.

[0131] The embodiment of the present application also provides a processor, which executes a computer program to perform at least the method described above.

[0132] The storage medium can be implemented by any type of nonvolatile storage device, or a combination thereof. The nonvolatile memory can be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Ferromagnetic Random Access Memory (FRAM), a Flash memory, a magnetic surface storage, an optical disc, or a Compact Disc Read-Only Memory (CD-ROM). The magnetic surface storage can be a magnetic disc memory or a magnetic tape memory. The storage medium described in the embodiments of the present application is intended to include, but is not limited to, these and any other suitable type of memory.

[0133] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other manners. The described device embodiments are merely schematic, and the division of the units is merely a logical function division. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, or direct coupling or communication connection between the components can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or in other forms.

[0134] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0135] In addition, each functional unit in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in the form of hardware, or in the form of hardware plus software functional units.

[0136] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program performs the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disc or an optical disc and various storage medium capable of storing program codes.

[0137] Alternatively, the integrated unit of the present application can be stored in a computer readable storage medium if it is realized in the form of a software function module and sold or used as an independent product. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes a mobile storage device, a ROM, a RAM, a magnetic disc or an optical disc and various storage medium capable of storing program codes.

[0138] The methods disclosed in the several method embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0139] The features disclosed in the several product embodiments provided by the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0140] The features disclosed in the several method or device embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0141] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art, without departing from the concept of the present application, a number of equivalent substitutions or obvious modifications can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. A method for embedding and reconstructing text-image alignment style transfer, characterized in that: The following steps are involved: S1. Basic Model Construction: Using a diffusion model as the foundational architecture, the model gradually adds noise to image data through a forward pass, simulating the image's transition from a clear state to a completely noisy state. Denoising is then used to recover the original image from the noisy image, enabling the learning and encoding of image style features. S2. Attention-based style extraction: For reference images in the artwork dataset, a feature extraction network is used to extract input image embeddings to capture the visual features of the image. The image embedding is then further processed using the perceptron attention mechanism and the feed-forward network FFN to capture complex style details and generate a style embedding; Step S2 specifically includes: Generate an initial latent variable for the reference image, which is normalized to stabilize the training process; Expand the latent variables to match the batch size of the input images to maintain consistency when processing multiple images; Updating the latent variables through the perceptron attention mechanism enables the model to selectively focus on different parts of the input image based on the learnable latent variables; The updated latent variables are transformed nonlinearly through the feedforward network FFN to enhance the style encoding capability; Combining the output of the feedforward network and the updated latent variables, an embedding representing the style of the input image is generated for subsequent text-image alignment enhancement; S3. Text embedding generation: The text description is converted into text embeddings through a text encoder to guide the style transfer process; S4. Text-Image Alignment Enhancement: The style embedding is fused with the text embedding via a cross-attention mechanism to generate a multimodal embedding to optimize the interaction between text and image information. Step S4 specifically includes: The style embedding and text embedding are converted into query, key and value matrices respectively through a linear layer so that they can interact in a shared feature space; The attention weights are computed using the dot product of the query matrix and the key matrix, and are scaled to prevent gradient vanishing, thereby dynamically prioritizing different aspects of the textual prompt. Apply the softmax function to the raw attention scores to obtain normalized attention weights, ensuring that the sum of the weights is 1 for effective information integration; The value matrix is ​​weighted summed using normalized attention weights to generate a multimodal embedding that can more effectively capture the multimodal context of text and images; S5. Integration of explicit modulation and diffusion model: The multimodal embedding is fused with the style embedding via linear interpolation. The resulting image embedding is concatenated with the text embedding to form a complete cue embedding. This cue embedding is then integrated into the diffusion model to generate a diverse style image that matches the text description.

2. The method for embedding and reconstructing text-image alignment style transfer according to claim 1, characterized in that: The diffusion model uses U-Net to implement the denoising process and is trained through mean square error loss.

3. The method for embedding and reconstructing text-image alignment style transfer according to claim 1 or 2, characterized in that: In step S2, the visual part of the CLIP model is used as a feature extraction network to obtain an embedded representation of the input image as a reference basis for style extraction; the output of the perceptron attention mechanism is further processed through a position-by-position feed-forward network (FFN) to capture more fine-grained style features; the output of the FFN is combined with the updated latent variables to obtain the final style embedding.

4. The method for embedding and reconstructing text-image alignment style transfer according to claim 1 or 2, characterized in that: The feedforward network FFN includes at least two linear transformation steps for weighting latent variables, and introduces a nonlinear activation function between the two linear transformations to enhance the model's ability to learn complex features.

5. The method for embedding and reconstructing text-image alignment style transfer according to any one of claims 1 to 2, characterized in that: Step S5 specifically includes: The style embedding is fused with the multimodal embedding obtained by text-image alignment enhancement through linear interpolation to achieve a smooth transition and effective combination between the two embeddings. Use predefined constants to control the fusion ratio between style embedding and multimodal embedding, so as to adjust the flexibility of fusion according to different application scenarios and requirements; The fused image embedding is concatenated with the text hint embedding to form a complete hint embedding for image generation, which integrates the multimodal information of text and image. Integrating the generated complete prompt embedding into the diffusion model enables the model to simultaneously consider the multimodal conditions of text and image during the generation process, improving the performance and quality of generated images.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for embedding and reconstructing text-image alignment style transfer according to any one of claims 1 to 5 is implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for embedding and reconstructing text-image alignment style transfer according to any one of claims 1 to 5 is implemented.

Images