A method for generating style-aligned image sets based on Diffusion Transformer

Abstract

This invention proposes a style-aligned image set generation method based on Diffusion Transformer. The method first uses a text encoder to convert text prompts into text features, and then uses a customized DiT generation model, where the traditional multi-head self-attention layer is replaced by a style-aligned attention layer. This style-aligned attention layer uses text features to guide image generation, while using the first generated image as a style reference. Adaptive instance normalization (AdaIN) and a dot-product attention mechanism with translation scaling are used to ensure style consistency among the generated images. Ultimately, this method can efficiently generate style-consistent image sets in parallel, adapting to image generation needs with different aspect ratios, such as non-square images like posters.

Description

Technical Field

[0001] This invention relates to image generation technology, and in particular to a method for generating style-aligned image sets based on DiffusionTransformer. Background Technology

[0002] The task of generating style-consistent image sets falls under the branch of text-based image models in the fields of style transfer and personalization. Personalization methods for text-based image models generally involve fine-tuning on a given reference image or adjusting the attention layer of the text-based image model to fuse style features and content information without fine-tuning.

[0003] Some methods require fine-tuning of model parameters based on given style images. For example, StyleDrop relies on fine-tuning of a lightweight adapter layer at the end of each attention block in a non-autoregressive text-to-image converter. This can be achieved by training the adapter layer on a set of images sharing the same style to generate a consistent set of images. However, when trained on individual images, it struggles to generate a consistent set of images with diverse content. These methods require fine-tuning, are computationally expensive, and necessitate preparing a corresponding reference style set for training each style.

[0004] Another approach requires no fine-tuning; it adjusts the attention layer of the text-generated image model. For example, SwappingSelf-Attention fuses style and content information by replacing the target keys and values ​​in the self-attention layer with those in the reference style image. InstantStyle injects reference style features into specific cross-attention layers of the IP-Adapter, decoupling style and content in the feature space from the reference image. The closest approach is StyleAligned, where the reference style image and descriptive textual cues are used to extract style features via DDIM inversion. Subsequently, the target query and key are normalized using an adaptive instance normalization formula based on the reference correspondence. Finally, the reference image keys and values ​​are merged with the DDIM inversion latent vectors in the self-attention layer. This method uses a UNet-based model, requiring a large number of parameters and long inference time to achieve good results. When using a lightweight UNet model, good style consistency and text alignment cannot be achieved, especially when generating high-resolution images and unconventional square-sized images.

[0005] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The main objective of this invention is to solve the problems existing in the above-mentioned background technology and provide a method for generating style-aligned image sets based on Diffusion Transformer.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for generating style-aligned image sets based on Diffusion Transformer, comprising:

[0009] Receive a set of text prompts, which describe the style and content features of the image set to be generated; and use a text encoder module to encode the text prompts into text features;

[0010] A DiT generative model based on Diffusion Transformer is constructed and used. This model generates images through a series of Transformer modules, where the original multi-head self-attention layer of DiT is replaced with a style-aligned attention layer. First, randomly initialized latent vectors are input into the first Transformer module of the DiT model. In each subsequent Transformer module of the DiT generative model, the latent vectors output by the previous module are processed by the style-aligned self-attention layer, and the processed latent vectors are passed as queries to the next style-aligned attention layer. In the style-aligned attention layer, the text features obtained in step S1 are injected as keys and values ​​to guide the image generation process. Through the processing of each Transformer module, the latent vectors are output in the last layer of the DiT generative model, serving as the basis for the generated image.

[0011] The DiT generation model uses the first generated image as a style reference through a style alignment attention layer and performs adaptive instance normalization (AdaIN) on the latent vectors of all generated images to align the style. It uses a displacement-scaled dot product attention mechanism to fuse style information in the style alignment attention layer. Thus, by sharing and aligning style information through the style alignment attention layer, it achieves parallel generation of multiple images to generate a style-consistent image set.

[0012] Furthermore, the text encoder module is a T5 model.

[0013] Furthermore, the DiT generative model forms a deep network architecture by stacking 28 Transformer modules. Each Transformer module gradually extracts multi-level features of the image through a self-attention mechanism and a feedforward network.

[0014] Furthermore, the operation of the style alignment attention layer includes:

[0015] The generated latent vector of the image is projected onto the query Q, key K, and value V vectors;

[0016] Using the corresponding components of the first generated image as a reference, adaptive instance normalization AdaIN is applied to the latent vectors of all generated images to adjust the query and key vectors.

[0017] The normalized query and key are concatenated with the reference key and value to form a fused style feature;

[0018] By employing a displacement-scaling dot product attention mechanism combined with a shared fractional variation factor, information fusion is performed on style features.

[0019] Adjust the value of the adjustable parameter X to balance image style consistency and text alignment.

[0020] Furthermore, the method includes the following operations:

[0021] Use the latent vectors of the first generated image as a style reference;

[0022] For the latent vectors of each subsequently generated image, adaptive instance normalization AdaIN is performed to align the style with the style reference latent vector.

[0023] The style reference latent vector and the target latent vector are fused to form a style-aligned attention layer input;

[0024] A dot product attention mechanism with displacement scaling is applied, combined with style reference and target latent vector, to achieve the fusion of style information;

[0025] By adjusting the shared score variation factor S, which is determined by the natural logarithm of the set adjustable parameter X, the balance between style consistency and text alignment is controlled.

[0026] Multiple latent vectors are processed in parallel to generate a style-consistent set of images.

[0027] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the described Diffusion Transformer-based style-aligned image set generation method.

[0028] A computer program product includes a computer program that, when executed by a processor, implements the Diffusion Transformer-based style-aligned image set generation method.

[0029] The present invention has the following beneficial effects:

[0030] This invention proposes a fine-tuning-free scheme using a diffusion model based on Diffusion Transformer (DiT), and introduces a novel framework, SAGA-DT, for the generation of style-consistent image sets. It achieves good quantitative and qualitative results on a lightweight DiT-based model, demonstrating sufficient efficiency in terms of model parameters and inference time. This invention excels in supporting the efficient generation of style-consistent images with arbitrary aspect ratios, adapting to more scenarios such as the generation of non-square images like poster sizes, and exhibits good generalization ability.

[0031] Compared with traditional methods, the main advantages of this invention are:

[0032] (1) Previous similar methods were all based on the UNet framework. This invention introduces Diffusion Transformer into style consistency generation for the first time, integrates the style alignment attention module into the DiT generation model, and outperforms other UNet-based methods with the same parameter scale in terms of style consistency and text alignment. It demonstrates for the first time the potential of Diffusion Transformer in supporting image style alignment.

[0033] (2) This method has outstanding performance in supporting the efficient generation of style-consistent images with arbitrary aspect ratios. It can adapt to more scenarios, such as the generation of non-square images like poster sizes, and has good generalization ability.

[0034] (3) Traditional algorithms have long inference time and a large number of model parameters, while the method of this invention is free from training and fine-tuning, and is efficient enough in terms of model parameters and inference time, thus achieving high-efficiency style alignment generation.

[0035] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0036] Figure 1 This is a flowchart of a style-aligned image set generation method based on Diffusion Transformer according to an embodiment of the present invention.

[0037] Figure 2 This is a diagram illustrating the style-aligned image set generation effect based on Diffusion Transformer in an embodiment of the present invention.

[0038] Figure 3 This is a comparison diagram of the qualitative effect of square images in an embodiment of the present invention.

[0039] Figure 4 This is a comparison diagram of the qualitative effects of non-square images in an embodiment of the present invention.

[0040] Figure 5This is a qualitative result of an ablation experiment regarding the shared fractional transfer factor in an embodiment of the present invention.

[0041] Figure 6 This is the quantitative result of the ablation experiment regarding the shared fractional transfer factor in the embodiments of the present invention. Detailed Implementation

[0042] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0043] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0044] This invention proposes a style-aligned image set generation method, SAGA-DT, based on Diffusion Transformer. This method first uses a text encoder to convert text prompts into text features, and then uses a customized DiT generation model, where the traditional multi-head self-attention layer is replaced by a style-aligned attention layer. The style-aligned attention layer uses text features to guide image generation, while using the first generated image as a style reference. Adaptive instance normalization (AdaIN) and a dot-product attention mechanism with translation scaling are used to ensure style consistency among the generated images. Ultimately, this method can efficiently generate style-consistent image sets in parallel, adapting to the generation needs of images with different aspect ratios, such as non-square images like posters.

[0045] See Figure 1 This invention provides a method for generating style-aligned image sets based on Diffusion Transformer, comprising:

[0046] Receive a set of text prompts, which describe the style and content features of the image set to be generated, and use a text encoder module to encode the text prompts into text features;

[0047] A DiT generative model based on Diffusion Transformer is constructed and used. This model generates images through a series of Transformer modules, replacing the multi-head self-attention layer in DiT with a style-aligned attention layer. First, randomly initialized latent vectors are input into the first Transformer module of the DiT model. In each subsequent Transformer module of the DiT generative model, the latent vectors output by the previous module are processed by the style-aligned self-attention layer, and the processed latent vectors are passed as queries to the next style-aligned attention layer. In the style-aligned attention layer, the text features obtained in step S1 are injected as keys and values ​​to guide the image generation process. Through the processing of all Transformer modules, the latent vectors are output in the last layer of the DiT generative model, serving as the basis for the generated images.

[0048] While the DiT generative model generates the latent representation of each image, the first generated image is used as a style reference through a style alignment attention layer, and adaptive instance normalization AdaIN is applied to the latent vectors of all generated images to align the style. A dot product attention mechanism with translation scaling is used to fuse style information in the style alignment attention layer. Thus, by sharing and aligning style information through the style alignment attention layer, multiple images can be generated in parallel to generate a set of images with consistent style.

[0049] In a preferred embodiment, the operation of the style alignment attention layer includes: projecting the generated image latent vectors onto query Q, key K, and value V vectors; using the corresponding components of the first generated image as a reference, performing adaptive instance normalization (AdaIN) on the latent vectors of all generated images to adjust the query and key vectors; concatenating the normalized query and key with the reference key and value to form fused style features; performing information fusion on the style features through a displacement-scaling dot product attention mechanism combined with a shared fraction change factor; and balancing image style consistency and text alignment by adjusting the value of a set adjustable parameter X.

[0050] See Figure 1 In a preferred embodiment, the method includes the following operations: using the latent vector of the first generated image as a style reference; performing adaptive instance normalization (AdaIN) on the latent vector of each subsequently generated image to perform style alignment based on the style reference latent vector; fusing the style reference latent vector with the target latent vector to form a style-aligned attention layer input; applying a displacement-scaled dot product attention mechanism to combine the style reference and the target latent vector to achieve style information fusion; controlling the balance between style consistency and text alignment by adjusting the shared score variation factor S, which is determined by the natural logarithm of a set adjustable parameter X; and processing multiple latent vectors in parallel to generate a style-consistent image set.

[0051] This invention proposes a style-aligned image set generation method based on Diffusion Transformer, and designs an innovative text-based image model called SAGA-DT to generate style-aligned image sets. This method introduces a style-alignment attention layer, replacing the multi-head self-attention layer in the traditional Transformer model, achieving a breakthrough in generating images with consistent style without fine-tuning. Compared with previous UNet-based methods, this invention exhibits significant advantages in style consistency and text alignment, while eliminating the need for training and fine-tuning, greatly reducing the number of model parameters and inference time, and improving the efficiency of image style-alignment generation.

[0052] Another significant advantage of this invention is its ability to generate images with arbitrary aspect ratios, making it adaptable to a wider range of applications, such as posters, advertisements, and personalized designs. Furthermore, this method maintains excellent style consistency and text alignment even when generating high-resolution and unconventional square-sized images, solving the problems of inconsistent style and incomplete content display commonly found in previous technologies when handling non-standard image sizes.

[0053] Furthermore, this invention validates the effectiveness of the style alignment attention layer through ablation experiments and demonstrates the flexibility of balancing style consistency and text alignment by adjusting the shared score variation factor. This balance provides artists and designers with greater creative freedom, enabling them to generate style-consistent and content-matched image sets according to different application needs.

[0054] The invention has a very broad application prospect. It can be used not only in the fields of artistic creation and design to improve creative efficiency and creative expression, but also in multiple fields such as advertising and marketing and personalized product development, providing users with a wide variety of design options, enhancing user experience, and promoting innovation and development in related industries.

[0055] The following further describes algorithm examples and experimental verifications of specific embodiments of the present invention.

[0056] Given a set of text prompts, the goal of style-aligned image generation is to generate a set of images with the same style while faithfully preserving the content described by the text prompts. This invention proposes a novel fine-tuning-free method using a diffusion model based on DiFfusion Transformer. This method shares style information among parallel-generated images by replacing the self-attention layer based on the DiT model with a plug-and-play style-alignment attention layer. With the same parameter scale and fewer main model parameters, this invention achieves superior performance both qualitatively and quantitatively. Figure 1This is an overall flowchart of the style-aligned image set generation method based on Diffusion Transformer according to an embodiment of the present invention.

[0057] (1) Text-based graph model based on DiT

[0058] Previous style alignment methods primarily employed U-Net-based diffusion models. State-of-the-art U-Net-based diffusion models contain a large number of parameters, resulting in long inference times for image generation. Furthermore, for lightweight U-Net models, previous methods exhibited poor style consistency and text alignment, particularly when generating non-square images with unusual aspect ratios, often resulting in incomplete content display and failing to adapt well to various image scales. To efficiently generate high-quality style-aligned image sets, this invention introduces a Transformer-based diffusion model, DiT. By fully utilizing the advantages of the DiT model, this invention surpasses previous U-Net-based models in both qualitative and quantitative aspects.

[0059] The DiT-based generative model in this invention consists of 28 Transformer modules. Compared to conventional DiT models, a multi-head cross-attention layer is introduced to inject textual information. Given textual cues, textual features are extracted by a T5 text encoder and then passed as keys and values ​​to the multi-head cross-attention layer to guide the image generation process. For each Transformer module, the latent vector is first processed by a multi-head self-attention layer and passed as a query to the multi-head cross-attention layer. This query interacts with keys and values ​​representing the extracted textual features and is subsequently passed through the feedforward layer. During this process, the input latent vector of the first module is randomly initialized, while the input of each subsequent module is the output of the previous module. This process is repeated in all 28 Transformer modules until the final denoising result is obtained.

[0060] To achieve style alignment between images, this invention introduces a plug-and-play style alignment attention module and modifies the multi-head cross-attention layer of the DiT structure. The technical solution is described in detail below.

[0061] (2) Introduction of Style Alignment Attention Module

[0062] Given a set of cues with the same style description, existing text-to-image models can generate a set of images with the same style. However, differences in the interpretation of style descriptions can still lead to inconsistencies in style features, thus limiting the degree of style consistency. Inspired by the Style-Aligned method, this invention replaces the multi-head attention layer in DiT with a style-aligned attention layer to integrate style information between images, thereby generating a style-aligned image set in parallel.

[0063] The style alignment attention layer uses the first generated image as a style reference, and this reference information is shared among all concurrently generated images through the following process: First, the latent vectors of all generated images are projected into Q, K, and V, simultaneously serving as the target Qt, Kt, and Vt. Simultaneously, the corresponding components of the first generated image in Q, K, and V are used as references, Qr, Kr, and Vr, respectively. To align the target image with the reference image, Adaptive Instance Normalization (AdaIN) is applied to modify the target query Qt and key Kt based on Qr and Kr:

[0064]

[0065]

[0066] Next, the reference key is concatenated with the normalized target key to form the Krt, and the reference value is concatenated with the target value to form the value feature Vrt:

[0067]

[0068] Finally, shift-scaled dot product attention is used instead of standard-scaled dot product attention for information fusion in this attention layer:

[0069]

[0070] Where S represents the shared score variation factor, used to adjust the attention score:

[0071] S = ln(X).

[0072] The value of parameter X can be adjusted to control the balance between image style consistency and text alignment.

[0073] By combining a DiT-based diffusion model with a style-aligned attention layer, this invention proposes a novel, fine-tuning-free framework, SAGA-DT, for style-aligned image set generation, demonstrating excellent generalization ability while ensuring generation efficiency. SAGA-DT operates on a zero-shot basis, exhibits good versatility and low computational cost, showcasing the capabilities of DiffusionTransformers in style-aligned image generation.

[0074] experiment

[0075] Figure 2 This is an example of the style-aligned image set generation effect based on Diffusion Transformer, and the image includes a high-resolution image as well as an unconventional square image.

[0076] (1) Qualitative effect

[0077] This invention underwent a qualitative comparison with other methods, including ELITE, BLIP Diffusion, IP-Adapter, Dreambooth-LoRA, and StyleAligned. A pre-trained SDv1.5 model was used for the IP-Adapter, while an SDv1.4 model was used for the other methods. Figure 3 As shown, although the IP-Adapter, BLIP-Diffusion, and ELITE generate images with a consistent style across the image set, the generated objects do not match the given text cues. Furthermore, StyleAligned generates images with a consistent style and correct objects, but fails to accurately interpret the style described by the text cues. In contrast, this invention ensures style consistency across the generated image set and aligns the generated objects and styles with the text cues, regardless of whether the generated images are square or not. Figure 4 As shown.

[0078] (2) Quantitative effect

[0079] For style-aligned image set generation, the goal is to generate a set of images with a consistent style while adhering to the content text instructions. Therefore, for each set of generated images, this invention is compared with other methods in the following three aspects, and the results are shown in Table 1.

[0080] 1) Text alignment score: Calculate the CLIP text score between the generated image and its corresponding text cue to measure the text alignment ability of each method. The higher the score, the higher the consistency between the image and the text.

[0081] 2) Image distance score: Calculate the DINO distance score to evaluate the style alignment performance within the image set. The lower the score, the smaller the style difference between images, i.e., the higher the style consistency.

[0082] 3) Inference Time: The inference time required for each method to generate images was measured to evaluate its efficiency. As shown in Table 1, the present invention outperforms all compared methods in both text alignment score and inter-image distance score, and is only 0.6 s slower than the fastest BLIP-Diffusion, demonstrating a balance between performance and efficiency.

[0083] Table 1 Quantitative Comparison and Ablation Comparison

[0084]

[0085] (3) Ablation test

[0086] To verify the effectiveness of this invention, an analysis was first conducted without the Style Alignment Attention layer (w / o SAA). As shown in Table 1, although the model's text alignment is slightly improved in this case, its performance on the inter-image distance score is significantly reduced. Next, ablation experiments were performed with adaptive instance normalization within the Style Alignment Attention layer (w / o AdaIn). In this case, the model's performance on both metrics is significantly reduced. Furthermore, the introduction of the Style Alignment Attention layer introduces a slight additional inference cost. Visualizations of these experiments are shown in [link to visualization]. Figure 3 .

[0087] To evaluate the impact of the shared score transfer factor X on the generated results, the results are first presented under different X values, such as... Figure 5 As shown, when X is large, the generated images become more stylistically consistent, but this consistency comes at the cost of reduced alignment between text and images. Conversely, as X decreases, the style of the image set becomes less consistent, but the images are more faithful to the text descriptions. We further performed a quantitative analysis of X, and the results are shown in [Figure number missing]. Figure 6 In this invention, a larger X performs better in the inter-image distance score, but significantly worse in the text alignment score. Conversely, a smaller X leads to better text alignment, but worse in the metric of style consistency. Since the task of this invention prioritizes generating style-consistent results, X = 0.3 was chosen as the parameter for the final model to strike a balance between image style consistency and text cue alignment.

[0088] This invention can be applied to the generation of style-aligned image sets, enabling more generalized functionality. By adding DDIM inversion, it allows the generation of an image set highly aligned with a specified image style, improving the level of style transfer and personalization in text-based image models. Furthermore, considering the fine-grained loss caused by traditional DDIM inversion, a modified DDIM inversion can be used.

[0089] The self-attention layer of DiT can be further adjusted to achieve more effective and efficient style information fusion.

[0090] By adding a decoupling scheme for style features and content information, better style consistency can be achieved without compromising text alignment, drawing inspiration from implementations in the field of image editing.

[0091] The main advantages of this invention compared to traditional methods are:

[0092] (1) Previous similar methods were all based on the UNet framework. This invention introduces Diffusion Transformer into style consistency generation for the first time, integrates the style alignment attention module into the DiT generation model, and outperforms other UNet-based methods with the same parameter scale in terms of style consistency and text alignment. It demonstrates for the first time the potential of Diffusion Transformer in supporting image style alignment.

[0093] (2) This method has outstanding performance in supporting the efficient generation of style-consistent images with arbitrary aspect ratios. It can adapt to more scenarios, such as the generation of non-square images like poster sizes, and has good generalization ability.

[0094] (3) Traditional algorithms have long inference time and a large number of model parameters, while the method of this invention is free from training and fine-tuning, and is efficient enough in terms of model parameters and inference time, thus achieving high-efficiency style alignment generation.

[0095] This invention has broad application prospects and high practical value, for example:

[0096] (1) In the field of art and design, style-aligned image set generation functions can be integrated into Photoshop or Illustrator to help designers generate design drafts in multiple styles. This can improve design efficiency, reduce creation time, and enhance creative expression, potentially leading to an increase in user numbers.

[0097] (2) In the stationery design industry, style alignment technology can be used to generate diverse style series of stationery products, accelerate product development cycle and reduce design costs;

[0098] (3) In the field of advertising and marketing, the style-aligned image set generation function can be integrated into its online design platform to provide users with a variety of design options, enhance the user's creative experience, attract more users, increase user usage, and thus increase revenue;

[0099] (4) In the field of AIGC academic research, the highly style-consistent image set generated by this invention can be used as a reference image set for style transfer methods based on fine-tuning, avoiding the complicated process of manually selecting image sets.

[0100] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0101] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0102] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0103] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk drive or magnetic tape drive. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0104] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0105] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0106] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0107] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0109] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0110] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0111] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0112] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A method for generating style-aligned image sets based on Diffusion Transformer, characterized in that, include: Receive a set of text prompts, which describe the style and content characteristics of the image set to be generated; The text prompt is then encoded into text features using a text encoder module. A DiFfusion Transformer-based DiT generative model is constructed and used. This model generates images through a series of Transformer modules, where the original multi-head self-attention layer of DiT is replaced with a style-aligned attention layer. First, randomly initialized latent vectors are input into the first Transformer module of the DiT model. In each subsequent Transformer module of the DiT generative model, the latent vectors output by the previous module are processed by the style-aligned self-attention layer, and the processed latent vectors are passed as queries to the next style-aligned attention layer. In the style-aligned attention layer, textual features are injected as keys and values ​​to guide the image generation process. Through the processing of each Transformer module, the latent vectors are output in the final layer of the DiT generative model, serving as the basis for the generated image. The DiT generative model uses the first generated image as a style reference through a style alignment attention layer, and performs adaptive instance normalization (AdaIN) on the latent vectors of all generated images to align the style. It then uses a displacement-scaled dot-product attention mechanism to fuse style information within the style alignment attention layer. Thus, by sharing and aligning style information through the style alignment attention layer, it achieves parallel generation of multiple images to generate a style-consistent image set. The operations of the style alignment attention layer include: projecting the generated image latent vectors onto query Q, key K, and value V vectors; using the corresponding components of the first generated image as a reference, performing adaptive instance normalization (AdaIN) on the latent vectors of all generated images to adjust the query and key vectors; concatenating the normalized query and key with the reference key and value to form fused style features; fusing style features through a displacement-scaled dot-product attention mechanism combined with a shared score variation factor; and adjusting the value of a set adjustable parameter X to balance image style consistency and text alignment.

2. The style-aligned image set generation method based on Diffusion Transformer as described in claim 1, characterized in that, The text encoder module is a T5 model.

3. The method for generating style-aligned image sets based on DiffusionTransformer as described in claim 1 or 2, characterized in that, The DiT generative model forms a deep network architecture by stacking 28 Transformer modules. Each Transformer module gradually extracts multi-level features of the image through a self-attention mechanism and a feedforward network.

4. The method for generating style-aligned image sets based on Diffusion Transformer as described in any one of claims 1 to 2, characterized in that, Includes the following operations: Use the latent vectors of the first generated image as a style reference; For the latent vectors of each subsequently generated image, adaptive instance normalization AdaIN is performed to align the style with the style reference latent vector. The style reference latent vector and the target latent vector are fused to form a style-aligned attention layer input; A dot product attention mechanism with displacement scaling is applied, combined with style reference and target latent vector, to achieve the fusion of style information; By adjusting the shared score variation factor S, which is determined by the natural logarithm of the set adjustable parameter X, the balance between style consistency and text alignment is controlled. Multiple latent vectors are processed in parallel to generate a style-consistent set of images.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the style-aligned image set generation method based on Diffusion Transformer as described in any one of claims 1 to 4.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the style-aligned image set generation method based on Diffusion Transformer as described in any one of claims 1 to 4.

Images