Method for generating style alignment image set based on Diffusion Transform

Abstract

The invention provides a style alignment image set generation method based on a Diffusion Transform. The style alignment image set generation method based on the Diffusion Transform comprises the following steps of: firstly, generating a style alignment image set; according to the method, firstly, a text encoder is used for converting text prompts into text features, then a model is generated through a customized DiT, and a traditional multi-head self-attention layer is replaced by a style alignment attention layer. The style alignment attention layer uses text features to guide image generation, uses the first generated image as a style reference, and ensures that the styles of the generated images are consistent through an adaptive instance normalization AdaIN and displacement scaling dot product attention mechanism. Finally, according to the method, image sets with consistent styles can be efficiently generated in parallel, and the method is suitable for image generation requirements of different aspect ratios, such as posters and other non-square images.

Description

Technical Field

[0001] The present invention relates to image generation technology, and in particular to a method for generating a style-aligned image set based on Diffusion Transformer. Background Art

[0002] The task of generating a set of images with consistent style belongs to the branch of style transfer and personalized customization of the text-based graph model. The personalization method of the text-based graph model generally integrates style features and content information without fine-tuning by fine-tuning on a given reference image or adjusting the attention layer of the text-based graph model.

[0003] Some methods require fine-tuning of model parameters based on a given style image. For example, StyleDrop relies on fine-tuning of a lightweight adapter layer at the end of each attention block in a non-autoregressive generative text-to-image converter, which can generate a set of images of the same style by training the adapter layer on a collection of images that share the same style. However, when training on a single image, it is difficult to generate a consistent set of images of different content. Such methods require fine-tuning, are computationally expensive, and require a corresponding reference style set to be prepared for each style for training.

[0004] Another type of method does not require fine-tuning, but adjusts the attention layer of the text-generated graph model. For example, SwappingSelf-Attention fuses style and content information by replacing the target keys and values ​​in the self-attention layer with those in the reference style image. InstantStyle injects reference style features into a specific cross-attention layer of IP-Adapter to decouple the style and content in the feature space from the reference image. Among them, the closest to this scheme is the StyleAligned technique, in which the reference style image and textual cues describing the style are used to extract style features through DDIM inversion. Subsequently, the target query and key are normalized using an adaptive instance norm formula based on the reference counterpart. Finally, the reference image key and value are merged with the DDIM inverted latent vector in the self-attention layer. This method uses a UNet-based model, which requires a large number of parameters and a long inference time to achieve good results. When using a lightweight UNet model, good style consistency and text alignment cannot be achieved, especially when generating high-resolution images and images of unconventional square sizes.

[0005] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0006] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and provide a method for generating a style-aligned image set based on Diffusion Transformer.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for generating a style-aligned image set based on Diffusion Transformer, comprising:

[0009] Receiving a set of textual prompts, the textual prompts are used to describe the style and content characteristics of the image set to be generated; and encoding the textual prompts into textual features using a text encoder module;

[0010] Construct and use a Diffusion Transformer-based DiT generative model, which generates images through a series of Transformer modules, in which the original multi-head self-attention layer of DiT is replaced by a style-aligned attention layer; wherein a randomly initialized latent vector is first input into the first Transformer module of the DiT model; in each subsequent Transformer module of the DiT generative model, the latent vector output by the previous module is processed by the style-aligned self-attention layer, and the processed latent vector is passed as a query to the next style-aligned attention layer; in the style-aligned attention layer, the text features obtained in step S1 are injected as keys and values ​​to guide the image generation process; through the processing of each Transformer module, the latent vector is output in the last layer of the DiT generative model as the basis for generating an image;

[0011] The DiT generation model uses the first generated image as a style reference through a style alignment attention layer, and performs adaptive instance normalization AdaIN on the latent vectors of all generated images to align the styles; a displacement-scaled dot product attention mechanism is used to fuse style information in the style alignment attention layer; thus, style information is shared and aligned through the style alignment attention layer to achieve parallel generation of multiple images to generate a set of images with consistent styles.

[0012] Furthermore, the text encoder module is a T5 model.

[0013] Furthermore, the DiT generation model forms a deep network architecture by stacking 28 Transformer modules, and each Transformer module gradually refines the multi-level features of the image through the self-attention mechanism and the feedforward network.

[0014] Furthermore, the operation of the style alignment attention layer includes:

[0015] Project the generated image latent vector to the query Q, key K, and value V vectors;

[0016] Adaptive instance normalization (AdaIN) is performed on the latent vectors of all generated images using the corresponding components of the first generated image as a reference to adjust the query and key vectors;

[0017] Concatenate the normalized query and key with the reference key and value to form a fused style feature;

[0018] The style features are fused through the displacement-scaled dot product attention mechanism combined with the shared score change factor;

[0019] Adjust the value of the adjustable parameter X to balance the image style consistency and text alignment.

[0020] Furthermore, the method comprises the following operations:

[0021] Use the latent vector of the first generated image as a style reference;

[0022] For the latent vector of each image generated subsequently, adaptive instance normalization AdaIN is performed to perform style alignment based on the style reference latent vector;

[0023] Fuse the style reference latent vector with the target latent vector to form the style-aligned attention layer input;

[0024] Applying the displacement-scaled dot-product attention mechanism to combine the style reference and target latent vectors to achieve the fusion of style information;

[0025] By adjusting the shared score variation factor S, which is determined by the natural logarithm of the set adjustable parameter X, the balance between style consistency and text alignment is controlled;

[0026] Multiple latent vectors are processed in parallel to generate a set of stylistically consistent images.

[0027] A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the method for generating a style-aligned image set based on a Diffusion Transformer is implemented.

[0028] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the method for generating a style-aligned image set based on a Diffusion Transformer is implemented.

[0029] The present invention has the following beneficial effects:

[0030] This paper proposes a fine-tuning-free scheme using a diffusion model based on Diffusion Transformer, and proposes a new framework SAGA-DT for the generation of style-consistent image sets. It achieves good quantitative and qualitative results on a lightweight model based on DiT, and is sufficiently efficient in terms of model parameters and inference time. This paper has outstanding performance in supporting the efficient generation of style-consistent images of arbitrary aspect ratios, and can adapt to more scenarios, such as the generation of non-square images such as poster sizes, with good generalization.

[0031] Compared with the traditional method, the main advantages of the present invention are:

[0032] (1) Previous similar methods were all based on the UNet framework. This paper first introduced the Diffusion Transformer into the style consistency generation work and integrated the style alignment attention module into the DiT generation model. It is superior to other existing UNet-based methods with the same parameter scale in terms of style consistency and text alignment, and for the first time demonstrated the potential of Diffusion Transformer in supporting image style alignment.

[0033] (2) This method has outstanding performance in supporting the efficient generation of style-consistent images with arbitrary aspect ratios. It can adapt to more scenarios, such as the generation of non-square images of poster size, and has good generalization performance.

[0034] (3) The traditional algorithm has a long reasoning time and a large number of model parameters, while the method of the present invention is free from training and fine-tuning, is sufficiently efficient in terms of model parameters and reasoning time, and achieves efficient style alignment generation.

[0035] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for generating a style-aligned image set based on a Diffusion Transformer according to an embodiment of the present invention.

[0037] Figure 2 This is an effect diagram of style alignment image set generation based on Diffusion Transformer in an embodiment of the present invention.

[0038] Figure 3 This is a comparison diagram of the qualitative effects of square images according to an embodiment of the present invention.

[0039] Figure 4 This is a comparison diagram of the qualitative effects of non-square images according to an embodiment of the present invention.

[0040] Figure 5The following are qualitative results of an ablation experiment on the shared fractional transfer factor according to an embodiment of the present invention.

[0041] Figure 6 The figure is a quantitative result of an ablation experiment on the shared fractional transfer factor according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0043] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0044] The present invention proposes a style-aligned image set generation method SAGA-DT based on Diffusion Transformer. The method first uses a text encoder to convert text prompts into text features, and then generates a model through a customized DiT, in which the traditional multi-head self-attention layer is replaced by a style-aligned attention layer. The style-aligned attention layer uses text features to guide image generation, and uses the first generated image as a style reference, and ensures that the generated image style is consistent through the dot product attention mechanism of adaptive instance normalization AdaIN and displacement scaling. Finally, the method can efficiently generate a set of images with consistent styles in parallel, adapting to the needs of image generation with different aspect ratios, such as non-square images such as posters.

[0045] See also Figure 1 The embodiment of the present invention provides a method for generating a style-aligned image set based on a Diffusion Transformer, comprising:

[0046] Receiving a set of textual prompts, the textual prompts are used to describe the style and content characteristics of the image set to be generated, and encoding the textual prompts into textual features using a text encoder module;

[0047] Construct and use the Diffusion Transformer-based DiT generative model, which generates images through a series of Transformer modules. The multi-head self-attention layer in DiT is replaced by the style-aligned attention layer. The randomly initialized latent vector is first input into the first Transformer module of the DiT model. In each subsequent Transformer module of the DiT generative model, the latent vector output by the previous module is processed by the style-aligned self-attention layer, and the processed latent vector is passed as a query to the next style-aligned attention layer. In the style-aligned attention layer, the text features obtained in step S1 are injected as keys and values ​​to guide the image generation process. After processing by all Transformer modules, the latent vector is output in the last layer of the DiT generative model as the basis for generating images.

[0048] While the DiT generative model generates the latent representation of each image, the first generated image is used as the style reference through the style alignment attention layer, and the latent vectors of all generated images are adaptively normalized AdaIN to align the styles; the style information is fused in the style alignment attention layer using the displacement-scaled dot product attention mechanism; thus, the style information is shared and aligned through the style alignment attention layer to achieve parallel generation of multiple images to generate a set of images with consistent style.

[0049] In a preferred embodiment, the operation of the style alignment attention layer includes: projecting the generated image latent vector to the query Q, key K and value V vectors; using the corresponding components of the first generated image as a reference, performing adaptive instance normalization AdaIN on the latent vectors of all generated images to adjust the query and key vectors; connecting the normalized query and key with the reference key and value to form a fused style feature; performing information fusion on the style feature through a displacement-scaled dot product attention mechanism combined with a shared score change factor; and balancing image style consistency and text alignment by adjusting the value of a set adjustable parameter X.

[0050] See also Figure 1 In a preferred embodiment, the method includes the following operations: using the latent vector of the first generated image as a style reference; performing adaptive instance normalization AdaIN for the latent vector of each subsequently generated image, and performing style alignment based on the style reference latent vector; fusing the style reference latent vector with the target latent vector to form a style-aligned attention layer input; applying a displacement-scaled dot-product attention mechanism to combine the style reference and target latent vectors to achieve fusion of style information; adjusting the shared score variation factor S, which is determined by the natural logarithm of the set adjustable parameter X, to control the balance between style consistency and text alignment; processing multiple latent vectors in parallel to generate a set of images with consistent styles.

[0051] This paper proposes a method for generating style-aligned image sets based on Diffusion Transformer and designs an innovative Diffusion Transformer-based style graph model, called SAGA-DT, for generating style-aligned image sets. This method introduces a style alignment attention layer. It replaces the multi-headed automatic Attention layer, which enables generation without fine-tuning A breakthrough in consistent style images. Compared with previous UNet-based methods, this invention shows significant advantages in style consistency and text alignment. It is also free from training and fine-tuning, greatly reducing the number of model parameters and inference time, and improving the efficiency of image style alignment generation.

[0052] Another significant advantage of the present invention is its ability to generate images of arbitrary aspect ratios, which enables it to adapt to more diverse application scenarios, such as posters, advertisements, personalized designs, etc. In addition, the method can still maintain excellent style consistency and text alignment when generating high-resolution and non-conventional square-sized images, solving the common problems of inconsistent style and incomplete content display in previous technologies when processing non-standard image sizes.

[0053] In addition, the present invention verifies the effectiveness of the style alignment attention layer through ablation experiments and demonstrates the flexibility of balancing style consistency and text alignment by adjusting the shared score variation factor. This balance provides artists and designers with more creative freedom, enabling them to generate image sets with consistent style and matching content according to different application requirements.

[0054] The application prospects of the present invention are very broad. It can not only be used in the fields of artistic creation and design to improve creative efficiency and creative expression, but also can be applied to advertising marketing, personalized product development and other fields to provide users with rich and diverse design options, enhance user experience, and promote the innovation and development of related industries.

[0055] The following further describes an algorithm example and experimental verification of a specific embodiment of the present invention.

[0056] Given a set of textual cues, the goal of style-aligned image generation is to generate a set of images of the same style while faithfully preserving the content described by the textual cues. We propose a novel fine-tuning-free approach using a diffusion model based on the Diffusion Transformer. This approach shares style information between parallel generated images by replacing the self-attention layer based on the DiT model with a plug-and-play style-aligned attention layer. We achieve superior performance both qualitatively and quantitatively with the same parameter scale and fewer main model parameters. Figure 1It is an overall flow chart of a method for generating a style-aligned image set based on a Diffusion Transformer according to an embodiment of the present invention.

[0057] (1) DiT-based text graph model

[0058] Previous style alignment methods were mainly implemented using a diffusion model based on U-Net. The most advanced diffusion model based on U-Net contains a large number of parameters and takes a long time to infer the generated images. For the lightweight U-Net model, the style consistency and text alignment of the generated images by the previous method were poor, especially when generating non-square images with special aspect ratios. In this case, the content is often not fully displayed, and it cannot adapt well to the generation of images of various proportions. In order to efficiently generate high-quality style-aligned image sets, this paper introduces a diffusion model DiT based on Transformer. Due to sufficient By utilizing the advantages of the DiT model, the present invention surpasses the previous The previous U-Net based model.

[0059] The DiT-based generative model in the present invention consists of a total of 28 Transformer modules. Compared with the conventional DiT model, a multi-head cross-attention layer is introduced to inject text information. The given text prompt is subjected to text feature extraction by the T5 text encoder, and then passed to the multi-head cross-attention layer as keys and values ​​to guide the image generation process. For each Transformer module, the latent vector is first processed by the multi-head self-attention layer and passed to the multi-head cross-attention layer as a query. This query interacts with the keys and values ​​representing the extracted text features and is subsequently passed through the forward layer. In this process, the input latent vector of the first module is randomly initialized, and the input of each subsequent module is the output of the previous module. The present invention repeats the above process in all 28 Transformer modules until the final denoising result is obtained.

[0060] In order to achieve style alignment between images, the present invention introduces a plug-and-play style alignment attention module and modifies the multi-head cross attention layer of the DiT structure. The technical solution is described in detail below.

[0061] (2) Introduction of style alignment attention module

[0062] Given a series of prompts with the same style description, existing text-to-image models can generate a set of images with the same style. However, differences in the way style descriptions are interpreted may still lead to inconsistencies in style features, thus limiting the degree of style consistency. Inspired by the Style-Aligned method, the present invention replaces the multi-head attention layer in DiT with a style-aligned attention layer to achieve the integration of style information between images, thereby generating a set of style-aligned images in parallel.

[0063] The style alignment attention layer takes the first generated image as the style reference, and this reference information is shared among all parallel generated images through the following process - First, the latent vectors of all generated images are projected into Q, K, and V, and used as targets Qt, Kt, and Vt. At the same time, the corresponding components of the first generated image in Q, K, and V are taken as references as Qr, Kr, and Vr, respectively. In order to align the target image with the reference image, adaptive instance normalization (AdaIN) is applied to modify the target query Qt and key Kt according to Qr and Kr:

[0064]

[0065]

[0066] Next, the reference key is concatenated with the normalized target key to form Krt, and the reference value is concatenated with the target value to form the value feature Vrt:

[0067]

[0068] Finally, use the shifted dot product attention instead of the standard scaled dot product attention to perform information fusion in this attention layer:

[0069]

[0070] Among them, S represents the sharing score change factor, which is used to adjust the attention score:

[0071] S=ln(X).

[0072] The value of parameter X can be adjusted to control the balance between image style consistency and text alignment.

[0073] By combining a DiT-based diffusion model with a style-aligned attention layer, we propose a novel, fine-tuning-free framework, SAGA-DT, for style-aligned image set generation, which demonstrates excellent generalization capabilities while ensuring generation efficiency. SAGA-DT operates in a zero-shot manner, has good versatility, and low computational cost, demonstrating the capabilities of DiffusionTransformers in style-aligned image generation.

[0074] experiment

[0075] Figure 2 An example of the effect of generating a style-aligned image set based on the Diffusion Transformer, which includes high-resolution images and unconventional square images.

[0076] (1) Qualitative effect

[0077] The present invention is qualitatively compared with other methods, including ELITE, BLIP Diffusion, IP-Adapter, Dreambooth-LoRA and StyleAligned. Among them, the pre-trained SDv1.5 model is used for IP-Adapter, while the SDv1.4 model is used for other methods. Figure 3 As shown in Figure 2, although the results generated by IP-Adapter, BLIP-Diffusion, and ELITE maintain a consistent style across image sets, the generated objects do not match the given text prompts. In addition, the results generated by StyleAligned have a consistent style and correct objects, but cannot accurately understand the style described by the text prompt. In contrast, the present invention ensures style consistency across generated image sets and aligns the generated objects and styles with the text prompts, regardless of whether the generated image is square or not, as shown in Figure 2. Figure 4 shown.

[0078] (2) Quantitative effect

[0079] For style-aligned image set generation, the goal is to generate a set of images with consistent style while following the content text instructions. Therefore, for each set of generated images, the present invention is compared with other methods from the following three aspects, and the results are shown in Table 1.

[0080] 1) Text alignment score: Calculate the CLIP between the generated image and its corresponding text cue Text score, in The text alignment ability of each method is measured. The higher the score, the better the image and text are aligned. The higher the consistency;

[0081] 2) Distance score between images: Calculate the DINO distance score to evaluate the style alignment performance within the image set. The lower the score, the smaller the style gap between images, that is, the higher the style consistency;

[0082] 3) Inference time: The inference time required for each method to generate an image is measured to evaluate its efficiency. As shown in Table 1, the proposed method outperforms all the compared methods in both text alignment score and inter-image distance score, and is only 0.6s slower than the fastest BLIP-Diffusion, demonstrating a balance between performance and efficiency.

[0083] Table 1 Quantitative comparison and ablation comparison

[0084]

[0085] (3) Ablation experiment

[0086] To verify the effectiveness of the proposed method, we first conduct an analysis without the style alignment attention layer (w / o SAA). As shown in Table 1, although the model's text alignment improves slightly in this case, its performance on the inter-image distance score drops significantly. Next, we conduct ablation experiments on the adaptive instance normalization within the style alignment attention layer (w / o AdaIn). In this case, the model's performance on both metrics drops significantly. In addition, the introduction of the style alignment attention layer brings a slight additional inference cost. The visualization of these experiments can be seen in Figure 3 .

[0087] In order to evaluate the impact of the shared score transfer factor X on the generated results, we first present the results under different X, such as Figure 5 As shown in , when X is larger, the style of the generated images becomes more consistent, but this consistency comes at the expense of reduced alignment between text and image. On the contrary, as X decreases, the style in the image set is less consistent, but the image is more faithful to the text description. We further perform a quantitative analysis of X and the results are shown in Figure 6 In the present invention, a larger X results in better performance in the inter-image distance score, but a significant decrease in the performance of the text alignment score. In contrast, a smaller X results in Better text alignment, but measured The index of style consistency is even worse. Considering the generation of style-consistent results, we chose X=0.3 as the parameter of the final model to strike a balance between image style consistency and text hint alignment.

[0088] The present invention can be applied to the generation of style-aligned image sets, can achieve more universal functions, add DDIM inversion, so that a set of image sets that are highly aligned with the style of a specified image can be generated, and the level of style transfer and personalization of the Wensheng graph model can be improved. At the same time, considering the fine-grained loss caused by traditional DDIM inversion, a DDIM inversion with correction can be used.

[0089] The self-attention layer of DiT can be further adjusted to achieve more effective and efficient style information fusion.

[0090] By adding a decoupling scheme for style features and content information, better style consistency generation can be achieved without compromising text alignment, which can be referenced in the field of image editing.

[0091] Compared with the traditional method, the main advantages of the present invention are:

[0092] (1) Previous similar methods were all based on the UNet framework. This paper first introduced the Diffusion Transformer into the style consistency generation work and integrated the style alignment attention module into the DiT generation model. It is superior to other existing UNet-based methods with the same parameter scale in terms of style consistency and text alignment, and for the first time demonstrated the potential of Diffusion Transformer in supporting image style alignment.

[0093] (2) This method has outstanding performance in supporting the efficient generation of style-consistent images with arbitrary aspect ratios. It can adapt to more scenarios, such as the generation of non-square images of poster size, and has good generalization performance.

[0094] (3) The traditional algorithm has a long reasoning time and a large number of model parameters, while the method of the present invention is free from training and fine-tuning, is sufficiently efficient in terms of model parameters and reasoning time, and achieves efficient style alignment generation.

[0095] The present invention has broad application prospects and high practical value, for example:

[0096] (1) In the field of art and design, the style alignment image set generation function can be integrated into Photoshop or Illustrator to help designers generate design drafts in various styles. This can improve design efficiency, reduce creation time, and enhance creative expression, which may lead to an increase in the number of users;

[0097] (2) In the stationery design industry, style alignment technology can be used to generate a variety of style series of stationery products, speed up the product development cycle, and reduce design costs;

[0098] (3) In the field of advertising and marketing, the style-aligned image set generation function can be integrated into its online design platform to provide users with a variety of design options, enhance the user creation experience, attract more users, increase user usage, and thus increase revenue;

[0099] (4) In the field of AIGC academic research, the highly style-consistent image set generated by the present invention can be For fine-tuning based Style transfer method, as a reference image set of stylized information, avoids artificial The complex process of screening image collections.

[0100] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0101] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0102] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0103] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0104] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0105] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0106] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be used as a processing unit. Elements are integrated into one unit; The integrated unit can be realized in the form of hardware orIt is implemented in the form of hardware plus software functional units.

[0107] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0108] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0109] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0110] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0111] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0112] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for generating style-aligned image sets based on Diffusion Transformer, characterized in that: include: Receiving a set of textual prompts, the textual prompts being used to describe the style and content characteristics of a desired image set to be generated; and encoding the text prompt into text features using a text encoder module; Construct and use a Diffusion Transformer-based DiT generative model, which generates images through a series of Transformer modules, in which the original multi-head self-attention layer of DiT is replaced by a style-aligned attention layer; wherein a randomly initialized latent vector is first input into the first Transformer module of the DiT model; in each subsequent Transformer module of the DiT generative model, the latent vector output by the previous module is processed by the style-aligned self-attention layer, and the processed latent vector is passed as a query to the next style-aligned attention layer; in the style-aligned attention layer, the text features obtained in step S1 are injected as keys and values ​​to guide the image generation process; through the processing of each Transformer module, the latent vector is output in the last layer of the DiT generative model as the basis for generating an image; The DiT generation model uses the first generated image as a style reference through a style alignment attention layer, and performs adaptive instance normalization AdaIN on the latent vectors of all generated images to align the styles; a displacement-scaled dot product attention mechanism is used to fuse style information in the style alignment attention layer; thus, style information is shared and aligned through the style alignment attention layer to achieve parallel generation of multiple images to generate a set of images with consistent styles.

2. The method for generating a style-aligned image set based on Diffusion Transformer according to claim 1, characterized in that: The text encoder module is a T5 model.

3. The method for generating a style-aligned image set based on DiffusionTransformer according to claim 1 or 2, characterized in that: The DiT generation model forms a deep network architecture by stacking 28 Transformer modules. Each Transformer module gradually refines the multi-level features of the image through the self-attention mechanism and feed-forward network.

4. The method for generating a style-aligned image set based on DiffusionTransformer according to any one of claims 1 to 3, characterized in that: The operation of the style alignment attention layer includes: Project the generated image latent vector to the query Q, key K, and value V vectors; Adaptive instance normalization (AdaIN) is performed on the latent vectors of all generated images using the corresponding components of the first generated image as a reference to adjust the query and key vectors; Concatenate the normalized query and key with the reference key and value to form a fused style feature; The style features are fused through the displacement-scaled dot product attention mechanism combined with the shared score change factor; Adjust the value of the adjustable parameter X to balance the image style consistency and text alignment.

5. The method for generating a style-aligned image set based on Diffusion Transformer according to any one of claims 1 to 4, characterized in that: The following operations are included: Use the latent vector of the first generated image as a style reference; For the latent vector of each image generated subsequently, adaptive instance normalization AdaIN is performed to perform style alignment based on the style reference latent vector; Fuse the style reference latent vector with the target latent vector to form the style-aligned attention layer input; Applying the displacement-scaled dot-product attention mechanism to combine the style reference and target latent vectors to achieve the fusion of style information; By adjusting the shared score variation factor S, which is determined by the natural logarithm of the set adjustable parameter X, the balance between style consistency and text alignment is controlled; Multiple latent vectors are processed in parallel to generate a set of stylistically consistent images.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a style-aligned image set based on Diffusion Transformer as described in any one of claims 1 to 5 is implemented.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating a style-aligned image set based on Diffusion Transformer as described in any one of claims 1 to 5 is implemented.

Images