A hybrid latent space and pixel space diffusion transformer image generation method

Abstract

The present application relates to the technical field of digital image generation, and particularly relates to a hybrid latent space and pixel space diffusion transformer image generation method, comprising: first, initializing conditional information and random noise latent variables; in each diffusion time step of the denoising trajectory, determining whether the current is in a high noise structure generation stage or a low noise detail recovery stage according to the noise level; in the high noise stage, inputting the current latent variable and the conditional information into a shared transformer backbone, outputting a latent space prediction result through a latent space branch, and updating the next step latent variable according to the latent space prediction result; in the low noise stage, restoring the current latent variable into a pixel space input image through a decoder, jointly injecting the pixel space input image and the current latent variable into the shared backbone after feature embedding, respectively, to perform double-flow joint feature extraction, outputting a latent space and a high-frequency residual prediction result, finally, summing and fusing the pixel input and the high-frequency residual prediction, and updating the latent variable in combination with the latent space prediction, until the iteration is completed to output a target image.

Description

Technical Field

[0001] This invention relates to the field of digital image generation technology, and in particular to an image generation method using a diffusion transformer that combines latent space and pixel space. Background Technology

[0002] Image generation technology aims to generate semantically accurate, structurally sound, and realistically detailed images based on given category labels or text descriptions. Currently, mainstream solutions are mainly based on diffusion models and diffusion transformers (DiT), and their technical approaches can be broadly divided into two categories: latent space diffusion models and pixel space diffusion models. Latent space diffusion models typically use a pre-trained variational autoencoder to compress the real image into a low-dimensional latent space for denoising sampling, significantly reducing computational overhead and improving training stability. Pixel space diffusion models, on the other hand, directly model the diffusion of pixels or image patches in the original image, without relying on autoencoder compression, and theoretically can retain more complete pixel-level high-frequency visual information.

[0003] Existing technologies present a clear inherent trade-off between the semantic efficiency of latent space generation and the detail fidelity of pixel space generation. Relying solely on latent space diffusion results in blurred textures and loss of high-frequency edge details during final image decoding due to spatial downsampling and limited capacity. On the other hand, relying solely on pixel space diffusion requires learning both global low-frequency structures and local high-frequency details simultaneously within the same network and at the same stage, which can easily lead to model capacity competition and target interference, resulting in an extremely heavy optimization burden. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a diffusion transform image generation method that combines latent space and pixel space, aiming to improve the problem that there is an inherent trade-off between semantic efficiency generated in latent space and detail fidelity generated in pixel space, which makes it impossible to obtain a stable global structure and clear high-frequency textures simultaneously under a unified framework.

[0005] This invention provides the following technical solution: a method for generating images using a diffusion transformer that combines latent space and pixel space, comprising: S1. Obtain conditional information and initialize random noise as the current latent variable in the latent space; S2. In each diffusion time step of the denoising trajectory, determine whether the current denoising trajectory is in the high noise structure generation stage or the low noise detail recovery stage based on the noise level at the current diffusion time step. S3. When in the high-noise structure generation stage, the current latent variables and condition information are input into the shared converter backbone, and the latent space prediction result corresponding to the current diffusion time step is output through the latent space branch. S4. When in the low-noise detail recovery stage, the current latent variable is restored through the decoder space of the variational autoencoder, and the corresponding pixel space input image is output. S5. Embed the pixel space input image and the current latent variable features respectively, and inject them together into the shared transformer backbone during the low noise detail recovery stage to perform dual-stream joint feature extraction, and output the latent space prediction result and the high-frequency residual prediction result respectively. S6. When in the high-noise structure generation stage, update the current latent variables in the next latent space according to the latent space prediction results; when in the low-noise detail recovery stage, sum and fuse the pixel space input image with the high-frequency residual prediction results, and update the current latent variables in the next latent space according to the fusion results and the latent space prediction results, until the denoising iteration is completed and the target image is output.

[0006] Preferably, in S1, the acquisition of condition information includes: The input text description is mapped to a conditional token sequence by a text encoder to obtain at least one of the input category label embedding vector or the input visual control signal. The visual control signal includes a visual control token obtained by feature encoding of edge signals, depth signals, or layout signals.

[0007] Preferably, in S2, determining whether the current denoising trajectory is in the high-noise structure generation stage or the low-noise detail recovery stage includes: Get the current diffusion time step and the preset time gating threshold; Determine whether the current diffusion time step is less than the preset time gating threshold; If the current diffusion time step is greater than or equal to the preset time gating threshold, then the current denoising trajectory is determined to be in the high-noise structure generation stage. If the current diffusion time step is less than the preset time gating threshold, the current denoising trajectory is determined to be in the low noise detail recovery stage.

[0008] Preferably, in step S4, the spatial downsampling ratio used by the decoder of the variational autoencoder to restore the current latent variable is consistent with the image block size used in step S5 when embedding features into the pixel space input image, so that the length of the pixel embedding sequence obtained by the division is the same as the length of the latent variable embedding sequence after the current latent variable space is expanded, and a corresponding arrangement relationship is established between the latent space position and the pixel space position.

[0009] Preferably, in S5, the joint injection of the shared converter backbone for dual-stream joint feature extraction during the low-noise detail recovery phase includes: In the low-noise detail recovery stage, a temporal gating injection mechanism is used to update the latent variable embedding sequence and the pixel embedding sequence obtained after feature embedding by performing feature gating superposition of the spatial hidden layer to obtain the fused feature sequence. The fused feature sequence is injected into the input of the shared transformer backbone, and dual-stream joint feature extraction is performed within the same transformer architecture.

[0010] Preferably, in S5, the output high-frequency residual prediction results include: The corresponding hidden states output by the shared transformer backbone are projected into spatial dimensions and rearranged into a two-dimensional feature grid. The two-dimensional feature grid is injected into the high-frequency pixel prediction head, and multiple cascaded convolutional thinning modules are used to progressively amplify and restore the spatial resolution, outputting the high-frequency residual prediction result.

[0011] Preferably, the stepwise scaling up and restoration of spatial resolution using multiple internally cascaded convolutional thinning modules includes: The two-dimensional feature grid is controlled to pass through the convolutional layer, sub-pixel convolutional upsampling layer, group normalization layer and activation function layer inside the convolutional thinning module in turn, and the feature grid is enlarged step by step by utilizing local feature integration and pixel rearrangement operations.

[0012] Preferably, the process of outputting high-frequency residual prediction results from the high-frequency pixel prediction head also includes a conditional modulation procedure: The global condition vector is obtained by summing the time step embeddings and the condition embeddings corresponding to the condition information. A multilayer perceptron is used to linearly map the global conditional vector to generate corresponding scaling and translation parameters. The scaling and translation parameters are used to perform adaptive layer normalization control on the feature mesh in the progressive magnification and restoration process.

[0013] Preferably, the internal network parameters of the shared transformer backbone, latent space branch, and pixel space branch are determined through the following joint optimization training process: At each full diffusion time step, the latent space velocity field flow matching loss corresponding to the latent space branch is calculated. When the current diffusion time step is less than the preset time gating threshold, the reconstruction loss and the perception loss are calculated by the mean square error operator and the pre-trained perceptual loss network respectively, and the pixel-level loss is obtained by weighted summation. The latent space velocity field flow matching loss and pixel-level loss are jointly constructed to form a total loss function, and the internal network parameters of the model are jointly optimized and fine-tuned based on the gradient of the total loss function through backpropagation.

[0014] Preferably, the sampling inference process for performing denoised trajectories includes the following steps: During the high-noise structure generation stage after the start of denoising sampling, only the latent space branch is activated for iterative denoising; When the denoising sampling iteration first enters the low-noise detail recovery stage, the decoder of the variational autoencoder is called to spatially restore the current latent variables and generate the initial pixel input of the pixel spatial input image. In each subsequent low-noise denoising sampling step, the current latent variable updated in the current step is injected together with the pixel space input image into the shared transformer backbone, so that the output high-frequency residual prediction result is synchronously refined along the latent space trajectory.

[0015] The present invention has the following beneficial effects: 1. In this invention, by setting a clear time-gating threshold and a control mechanism that allows pixel space branches to participate in calculations only in the low-noise stage, dynamic decoupling of the generation task in the two complementary stages of coarse-grained low-frequency semantic structure generation and fine-grained high-frequency pixel detail restoration is achieved. This solves the problem in traditional methods where the lack of time-stage division of labor in the denoising process leads to target conflict and network capacity competition between semantic structure generation and high-frequency texture restoration in the same network and the same stage.

[0016] 2. In this invention, by enabling the latent space branch and the pixel space branch to share the same multilayer diffusion transformer backbone network and combining the design scheme of equal spatial resolution mapping, the point-to-point cohesive interaction and global context depth utilization of the two branch features in a unified neural network base are realized. This solves the problems of network parameter expansion, excessive computational resource overhead, additional computational loss and serious overall semantic disconnect caused by simply deploying two independent models, as well as the problem of the introduction of complex alignment modules due to inconsistent cross-space feature lengths.

[0017] 3. In this invention, by rearranging the one-dimensional hidden state into a two-dimensional feature grid inside the high-frequency pixel prediction head, and using a topological architecture that performs progressive upsampling and adaptive layer normalization conditional modulation through cascaded multi-level convolution and sub-pixel convolution, the one-time regression task of high-dimensional large-size image blocks is decoupled into local continuous sub-block prediction with the introduction of local spatial inductive bias. This solves the problem that existing pixel spatial prediction heads, when using simple linear layers to directly regress high-dimensional image blocks, are prone to signal breakage, blurring, and severe visual artifacts in the recovered edges and micro-textures due to the lack of spatial local continuity. Attached Figure Description

[0018] Figure 1 This is a flowchart of an image generation method using a diffusion transformer that combines latent space and pixel space, as proposed in this invention. Figure 2This is a schematic diagram comparing the latent space diffusion, pixel space diffusion, and the hybrid latent space-pixel space diffusion framework of the image generation method using a diffusion transformer that combines latent space and pixel space proposed in this invention. Figure 3 This is a schematic diagram of the overall network framework of the Hi-DiT diffusion transform image generation method that combines latent space and pixel space proposed in this invention. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In embodiments of the present invention, a diffusion transform image generation method that combines latent space and pixel space is provided, such as... Figures 1-3 As shown, it includes the following steps: S1. Obtain conditional information and initialize random noise as the current latent variable in the latent space; Furthermore, in S1, obtaining conditional information includes: The input text description is mapped to a conditional token sequence by a text encoder to obtain at least one of the input category label embedding vector or the input visual control signal. Visual control signals include visual control tokens obtained by feature encoding of edge signals, depth signals, or layout signals.

[0021] Specifically, in the initial stage of image reasoning generation, the system first performs the acquisition of conditional information and the initialization of the latent space state, providing a basic physical basis and semantic guidance for the subsequent iteration of the denoising trajectory.

[0022] Conditional information plays a crucial role in establishing global semantic constraints and multimodal control throughout the generative network. To achieve generalized image generation under different modal inputs, the process of acquiring conditional information supports diverse input formats. When the input information is a text description, the text description is first fed into a pre-trained text encoder, which typically employs a contrastive language image pre-training model. Through its tokenizer and multi-layer self-attention mechanism, the text string is transformed into a continuous vector containing high-dimensional semantic features, i.e., a sequence of conditional tokens.

[0023] When the input information is a category label, the category label is directly mapped to a fixed-dimensional category label embedding vector through a learnable hidden layer embedding lookup table to provide coarse-grained scene semantic guidance. When precise image-to-image spatial control is required, the system acquires the input visual control signal. This visual control signal is first used by an operator or estimation network to extract the corresponding edge signal, depth signal, or layout signal. These signals are then input into a convolutional feature encoder for spatial feature channel mapping, ultimately transforming them into visual control tokens with the same channel dimension as the transformer backbone network. These conditional token sequences, category label embedding vectors, or at least one of the visual control tokens will subsequently serve as a conditional injection source to guide the directional convergence of the image content.

[0024] Simultaneously or subsequently, the system initializes random noise as the current latent variable in the latent space. This current latent variable serves as the state of the denoising trajectory at the zero initial time step, constructing a forward denoising trajectory based on a diffusion model. Specifically, the system utilizes a pseudo-random number generator to sample and generate initial Gaussian random noise according to a standard normal distribution, and defines it as the initial latent variable by combining the data distribution perturbation mechanism of the diffusion model. In the forward denoising or initial allocation process, the evolution from the real data distribution to the noise distribution is determined by the following linear interpolation trajectory formula: ; In the above linear interpolation trajectory formula, Indicates at any diffusion time step The potential state vector is given by: The range of values ​​is normalized to the continuous interval [0,1]. Represents noiseless, real-world latent variables; This represents standard Gaussian random noise in the sample. This occurs at the initial sampling point, i.e., time step... In the extreme state where the value equals 1, the first term in the formula is completely eliminated, and the potential state vector... Completely composed of Gaussian random noise The system then initializes the Gaussian random noise as the current latent variable in the latent space. This current latent variable serves as the initial canvas carrying all the potential structural information of the image to be constructed. In the subsequent denoising trajectory, it will continuously perform flow field velocity prediction and feature iteration in the feature space.

[0025] In actual system operation and application, the above steps achieve decoupling and representation alignment between multimodal control signals and random initial states. By uniformly transforming the inputs of different modalities into serialized tokens or embedding vectors, the subsequent shared transformer backbone network can simultaneously model the flow field vector field of the spatial distribution of the current latent variables and the semantic distribution of conditional information within a unified feature space. This operating mode ensures that the initialization phase possesses both sufficient random diversity and precise control determinism, thus laying a standard data input foundation for the dual-stream feature extraction process of hybrid latent space and pixel space.

[0026] S2. In each diffusion time step of the denoising trajectory, determine whether the current denoising trajectory is in the high noise structure generation stage or the low noise detail recovery stage based on the noise level at the current diffusion time step. Furthermore, in S2, determining whether the current denoising trajectory is in the high-noise structure generation stage or the low-noise detail recovery stage includes: Get the current diffusion time step and the preset time gating threshold; Determine whether the current diffusion time step is less than the preset time gating threshold; If the current diffusion time step is greater than or equal to the preset time gating threshold, then the current denoising trajectory is determined to be in the high-noise structure generation stage. If the current diffusion time step is less than the preset time gating threshold, the current denoising trajectory is determined to be in the low noise detail recovery stage.

[0027] Specifically, during the iterative advancement of the denoising trajectory, the system performs adaptive determination in the time domain at each diffusion time step, dividing the continuous denoising trajectory into two objective technical stages with different physical generation focuses, thereby providing a control benchmark for the subsequent feature stream splitting process.

[0028] The noise level at each diffusion time step is closely related to the temporal evolution of the denoising trajectory. The entire denoising trajectory consists of a series of discrete or continuous time sequences, and the noise level in the latent variables shows a linear decreasing trend as the time steps progress. To accurately capture this physical characteristic of the evolution from macroscopic structure to microscopic detail, the system introduces a time-gated injection mechanism. At the beginning of each diffusion time step, the system first obtains the current diffusion time step and a preset time-gated threshold. The current diffusion time step is a temporal scalar variable representing the current denoising progress, while the preset time-gated threshold is a critical scalar value pre-set within the time domain interval. Logically, this preset time-gated threshold divides the entire denoising trajectory into an early coarse-grained global structure shaping period and a later fine-grained high-frequency detail repair period.

[0029] After acquiring the two time-domain variables mentioned above, the system uses a digital comparator or logical judgment module to determine whether the current diffusion time step is less than a preset time-gating threshold. This judgment process is an objective pipeline operation based on time-domain coordinate comparison. If the current diffusion time step is greater than or equal to the preset time-gating threshold, the current denoising trajectory is determined to be in the high-noise structure generation stage. At this time, due to the high residual noise energy in the latent variables, the network mainly focuses on extracting the macroscopic geometric topology and global semantic framework from the large-scale disordered noise. If the current diffusion time step is less than the preset time-gating threshold, the current denoising trajectory is determined to be in the low-noise detail recovery stage. At this time, the basic structure of the latent variables has been solidified, and the residual noise mainly affects local textures and high-frequency edges. The network then shifts its computational focus to the precise compensation of microscopic details.

[0030] In actual system operation and application, this time-gated threshold-based stage determination manifests as an automatic triggering process of state control signals. To achieve seamless stage transitions and branch control within the network, the time-domain segmentation and state determination of the denoised trajectory are explicitly expressed using the following time-gated state function formula: ; In the above time-gated state function formula, Indicates the current diffusion time step The gated state control signal output is limited to binary zero or one; This represents the current diffusion time step, and its value is within a continuous interval from zero to one. This represents the preset time gating threshold, which is used as the critical time coordinate to distinguish between high and low noise levels. This indicates the indicator function operator. When the condition is met, i.e., at the current diffusion time step... Less than the preset time threshold When the indicator function operator is activated, the gated state control signal is output. If the value equals 1, the system determines that the current denoising trajectory is in the low-noise detail recovery stage, and thus automatically activates the variational autoencoder / decoder stream and the dual-stream joint feature extraction process in subsequent feature interactions. Conversely, when the gating state control signal is output... When the value equals zero, the system determines that the current denoising trajectory is in the high-noise structure generation stage and retains only the latent space feature mapping of a single stream. This operating mode achieves dynamic decoupling of the generation task under different noise levels through objective time-domain gating signals, providing deterministic control flow support for the full-process automated engineering implementation.

[0031] S3. When in the high-noise structure generation stage, input the current latent variables and condition information into the shared converter backbone, and output the latent space prediction result corresponding to the current diffusion time step through the latent space branch. Specifically, during the high-noise structure generation stage, the system initiates a single-stream latent space feature mapping mechanism, which sends the latent variables obtained from the initial allocation or the previous iteration update, along with the acquired conditional information, into the core computing network to perform large-scale global semantic and macro-topological structure prediction and construction.

[0032] When the current diffusion time step is determined to be greater than or equal to the preset time gating threshold based on the time gating state function, the gating state control signal output is zero. At this time, the system cuts off the activation path of the pixel space branch at the underlying logic control level, so that the computational topology of the entire network focuses on the pure latent space feature transformation flow. At the data flow level, the current latent variable, as the current input source, is first aligned in feature dimension. The aligned current latent variable and the acquired conditional information are injected into the shared transformer backbone network. This shared transformer backbone network is composed of a multi-layer bidirectional self-attention mechanism architecture. The transformer blocks inside it are completely parameter-shared across all diffusion time steps and all feature space branches, and the network weights do not change with the time step.

[0033] After entering the shared transformer backbone network, the spatial token sequence of the current latent variable and the conditional token sequence of the conditional information perform global feature interaction within the same hidden layer feature space. In each shared transformer block, the network calculates the association weights between tokens through multi-head self-attention layers, enabling the conditional information to guide the spatial disorder noise in a directional manner. After deep feature extraction and nonlinear transformation through all overlapping stacked layers of the shared transformer backbone network, the backbone outputs a hidden state sequence incorporating the conditional features. To transform the hidden state sequence into a velocity field asset that can guide the next time step iteration, the hidden state sequence is then fed into the immediately following latent space branch, where flow field vector regression is performed using the following forward mapping prediction formula: ; In the above forward mapping prediction formula, This represents the latent space prediction result corresponding to the current diffusion time step in the final output of the system. It represents the regression velocity field vector defined under the latent space flow matching theory, which is used to indicate the rate of change of latent variables as they evolve into noiseless real data. Represents the current latent variable in the input latent space; This indicates the input condition information; Indicates the current diffusion time step; This represents the deep self-attention feature extraction mapping performed by the shared transformer backbone network; This represents the linear projection and dimensional transformation function performed by the latent space branch. Through this forward computation, the system can eliminate cluttered components in random noise and extract the low-frequency velocity field that characterizes the general outline and color distribution of the target image.

[0034] In actual system operation and application, this forward mapping process achieves decoupled computation of the diffusion transformer in high-noise environments. By fully allocating computational resources to the convergence of global low-frequency semantics, the network can prioritize the establishment of stable and clear object shapes, color distributions, and scene structures in the upper-half denoising trajectory, avoiding the ineffective introduction and interference of local microscopic high-frequency details before the macroscopic structure is solidified. This operating mode ensures that the latent space prediction results can accurately capture the global skeleton of the image, thus providing an absolutely accurate and reliable structural foundation for dual-stream joint feature extraction and pixel-level high-frequency residual compensation when entering the low-noise detail restoration stage in later sampling.

[0035] S4. When in the low-noise detail recovery stage, the current latent variables are restored through the decoder space of the variational autoencoder, and the corresponding pixel space input image is output. Furthermore, in step S4, the spatial downsampling rate used by the decoder of the variational autoencoder to restore the current latent variable is consistent with the image block size used in step S5 when embedding features into the pixel space input image, so that the length of the pixel embedding sequence obtained by the division is the same as the length of the latent variable embedding sequence after the current latent variable space is expanded, and a corresponding arrangement relationship is established between the latent space position and the pixel space position.

[0036] Specifically, when the denoising trajectory sampling progresses to the low-noise detail recovery stage, the system initiates the inverse mapping flow from latent space to pixel space, restoring the high-dimensional latent features of the current step into an intuitive pixel-level representation, and establishing a spatial mapping benchmark between the two-stream features through strict geometric size alignment.

[0037] When the current diffusion time step is determined to be less than the preset time-gating threshold based on the time-gating state function, the gating state control signal outputs one, and the system immediately activates the pixel space branch. At the data flow level, the current latent variable is input into the decoder of the pre-trained variational autoencoder. The decoder performs spatial dimension amplification and restoration on the current latent variable through cascaded nonlinear transformation layers, thereby restoring the abstract latent features into a pixel space input image with color channels.

[0038] To ensure seamless fusion of latent space and pixel space features during subsequent dual-stream joint feature extraction, the system implements a strict spatial resolution alignment mechanism between the restoration process in step S4 and the embedding process in step S5. Specifically, the spatial downsampling rate used by the variational autoencoder to spatially restore the current latent variable is completely consistent with the image block size used for feature embedding of the pixel space input image later. Through this identical mapping, the length of the pixel embedding sequence obtained after the pixel space input image is segmented into non-overlapping image blocks and linearly projected is the same as the length of the latent variable embedding sequence after the current latent variable is spatially expanded and flattened. This geometric alignment establishes a one-to-one spatial arrangement relationship between the position coordinates in the latent space and the position coordinates in the pixel space, avoiding the introduction of additional alignment modules due to inconsistent lengths.

[0039] In actual system operation and application, in order to ensure that the underlying hardware pipeline of the entire image generation can accurately perform this cross-space sequence length alignment, the system explicitly constrains the transformation rules of geometric dimensions and data space during spatial restoration and sequence unrolling through image transformation and frequency decomposition formulas: ; ; In the above formula, Encoder mapping function representing variational autoencoder This represents the decoder restoration function of the variational autoencoder; Represents a real image; This represents the clean latent variable obtained after encoding; This represents the structural substrate image reconstructed from the decoder. It represents the low-frequency components of an image, which represent slowly changing visual structures such as overall layout, outline, main shape, and color blocks; This represents the high-frequency components of an image, indicating rapidly changing visual signals such as edges, textures, materials, and local details. The system reconstructs the structural substrate image from the decoder space. This is considered as support for the low-frequency structure, and the forward input construction of the pixel spatial branch is built on this basis to ensure that the pixel block sequence and the latent variable sequence satisfy a perfect balance in spatial resolution: ; In the above spatial resolution balance formula, and These represent the height and width of the input image in pixel space after being restored by the decoder; The side length of the image block used in step S5 for feature embedding of the pixel space input image is preferably set to sixteen, consistent with the downsampling factor of the variational autoencoder. and These represent the height and width of the current latent variable in the latent space, respectively. Due to the relationship between the downsampling rate and the size of the image patch... Complete equivalence ensures that the total number of pixel sequence tokens calculated on the left side of the equation is strictly equal to the total number of latent variable tokens after the plane expansion on the right side of the equation.

[0040] Based on the aforementioned size alignment, after the forward data flows of the two branches are flattened into a one-dimensional feature vector sequence, they exhibit point-to-point dual-flow collaboration on the same topological coordinates. The sequence index mapping and one-to-one correspondence arrangement of their hidden layers are expressed by the following sequence index mapping formula: ; In the above sequence index mapping formula, Indicates the index position of the latent variable embedded in the sequence. Hidden feature token; This indicates that the index position in the pixel embedding sequence is also... Pixel feature token; Represents the traversal index of a one-dimensional sequence; This indicates that both have identical spatial geometric and physical meanings in the joint modeling of the two streams in the shared transformer backbone network. This operation mode ensures that the high-frequency texture stream in pixel space and the macroscopic structure stream in latent space can directly perform feature-gated superposition interactions at the same sequence positions, establishing spatial correlations using the global context of the shared backbone network, and achieving accurate compensation for latent structure and autoencoder reconstruction errors by pixel-level micro-details.

[0041] S5. Embed the pixel space input image and the current latent variable features respectively, and inject them into the shared transformer backbone in the low noise detail recovery stage to perform dual-stream joint feature extraction, and output the latent space prediction result and the high-frequency residual prediction result respectively. Furthermore, in S5, the joint injection of the shared transformer backbone for dual-stream joint feature extraction during the low-noise detail recovery stage includes: In the low-noise detail recovery stage, a temporal gating injection mechanism is used to update the latent variable embedding sequence and the pixel embedding sequence obtained after feature embedding by performing feature gating superposition of the spatial hidden layer to obtain the fused feature sequence. The fused feature sequence is injected into the input of the shared transformer backbone, and two-stream joint feature extraction is performed within the same transformer architecture.

[0042] Furthermore, in S5, the output high-frequency residual prediction results include: The corresponding hidden states output by the shared transformer backbone are projected into spatial dimensions and rearranged into a two-dimensional feature grid. Two-dimensional feature meshes are injected into the high-frequency pixel prediction head, and multiple cascaded convolutional thinning modules are used to progressively amplify and restore the spatial resolution, outputting high-frequency residual prediction results.

[0043] Furthermore, the spatial resolution is progressively magnified and restored using multiple cascaded convolutional thinning modules, including: The two-dimensional feature grid is controlled to pass through the convolutional layer, sub-pixel convolutional upsampling layer, group normalization layer and activation function layer inside the convolutional thinning module in turn, and the feature grid is enlarged step by step by utilizing local feature integration and pixel rearrangement operations.

[0044] Furthermore, the process of outputting high-frequency residual prediction results from the high-frequency pixel prediction head also includes a conditional modulation procedure: The global condition vector is obtained by summing the time step embeddings and the condition embeddings corresponding to the condition information. A multilayer perceptron is used to linearly map the global conditional vector to generate corresponding scaling and translation parameters. By utilizing scaling and translation parameters, adaptive layer normalization control is applied to the feature mesh in the progressive magnification and restoration process.

[0045] Specifically, in the low-noise detail recovery stage of the denoising trajectory, the system activates the dual-stream joint feature extraction mechanism and the progressive high-frequency residual prediction process, enabling the network to focus on recovering lost local textures and high-frequency edges based on the structural anchors provided by the latent space.

[0046] The temporal gating injection mechanism is triggered when the system is in the low-noise detail recovery phase where the time step is less than a preset gating threshold. During the data input preparation phase, the system feeds the latent variable embedding sequence and pixel embedding sequence obtained after feature embedding into the spatial hidden layer. The system uses a temporal gating injection mechanism based on an indicator function to perform feature fusion, gating and overlaying two one-dimensional sequence tokens from different branches within the same hidden feature space. The multi-branch feature fusion process is explicitly calculated using the following hidden layer gating fusion formula: ; In the above hidden layer gated fusion formula, This represents the fused feature sequence obtained after fusion, which serves as the initial hidden state injected into the backbone network; This represents the sequence of latent variable embeddings after feature embedding; This represents the pixel embedding sequence after feature embedding; The binary control signal output represents the time-gated state function; This indicates the current diffusion time step. Since we are currently in a low-noise phase, the gated state control signal is output. Since the token lengths are strictly equal to one, the pixel embedding sequence is fully superimposed and updated into the latent features, achieving latent fusion of the two. Subsequently, the system injects the fused feature sequence into the input of the shared transformer backbone network. This backbone network adopts a transformer architecture with a multi-layer bidirectional self-attention mechanism. Because the token lengths of the latent variable sequence and the pixel sequence are strictly aligned, the backbone network directly performs dual-stream joint feature extraction within the same transformer architecture, enabling pixel features to utilize the global context formed by the latent space and avoiding conflicts between local texture and overall semantics.

[0047] After deep feature interaction via the shared transformer backbone network, the network outputs latent space prediction results and high-frequency residual prediction results through parallel prediction heads. The high-frequency residual prediction results are output by the high-frequency pixel prediction head of the pixel space branch. To avoid continuous signal breaks and artifacts caused by direct high-dimensional linear regression, the system first projects the corresponding hidden states output by the shared transformer backbone network into a spatial dimension and rearranges them into a two-dimensional feature mesh. Subsequently, the system injects this two-dimensional feature mesh into the high-frequency pixel prediction head, using multiple cascaded convolutional thinning modules to progressively amplify and restore the spatial resolution. The feature mesh is sequentially passed through convolutional layers, sub-pixel convolutional upsampling layers, group normalization layers, and activation function layers within the convolutional thinning modules. This progressive amplification and local rearrangement calculation process is expressed by the following convolutional thinning state operator formula: ; In the above formula for the convolution thinning state operator, Indicates the current number The refined feature grid output by the multi-level convolutional refinement module; This represents the feature mesh input from the previous module; This represents a standard two-dimensional convolution operation that introduces a local spatial inductive bias. This indicates a sub-pixel convolutional layer mapping that achieves spatial resolution upsampling through pixel rearrangement operations; Operators that perform group normalization operations; This represents the activation function layer that performs nonlinear activation. The system progressively amplifies the spatial resolution by a factor of four through two consecutive cascaded refinements, enabling the final linear output layer to output high-frequency residual prediction results by predicting only local sub-blocks.

[0048] In this process, to ensure absolute consistency between the high-frequency residual prediction results and the current denoising process and multimodal context, the high-frequency pixel prediction head introduces a conditional modulation procedure based on an adaptive layer normalization structure. The system sums the current time-step embedding with the conditional embedding corresponding to the conditional information to obtain the fused global conditional vector. Subsequently, the system uses a multilayer perceptron to linearly map this global conditional vector, generating corresponding scaling and translation parameters. The conditional parameter generation process is expressed by the following modulation parameter mapping formula: ; In the above modulation parameter mapping formula, This represents the global conditional vector obtained by adding the time-step embedding and the conditional embedding; Represents the linear and nonlinear projection functions performed by the multilayer perceptron; This represents the generated scaling parameter vector; This represents the generated translation parameter vector. After obtaining the scaling and translation parameters, the system injects them into the step-by-step magnification and restoration process, performing adaptive layer normalization control on the feature mesh. The control mechanism is explicitly calculated using the following adaptive layer normalization modulation formula: ; In the above adaptive layer normalized modulation formula... This represents the final output high-frequency residual prediction result, which is used to compensate for residual details in the reconstruction error of the variational autoencoder. This represents the final feature grid obtained after progressive upsampling through two levels of convolutional thinning modules; Indicates the standard layer normalization operator; Represents the dot product operator of the Hadamard product matrix; This represents the final linear projection output layer. This operating mode allows the generation of high-frequency details to be dynamically modulated by both denoising time and global semantic conditions, thereby ensuring high fidelity and consistency between the output high-frequency textures, edge contours, and overall content.

[0049] S6. When in the high-noise structure generation stage, update the current latent variables in the next latent space according to the latent space prediction results; when in the low-noise detail recovery stage, sum and fuse the pixel space input image with the high-frequency residual prediction results, and combine the fusion results with the latent space prediction results to update the current latent variables in the next latent space until the denoising iteration is completed and the target image is output.

[0050] Specifically, at the end of the denoising trajectory execution phase, the system dynamically switches the iterative update operator of the latent state according to the current time domain stage, and performs multimodal spatiotemporal fusion of the velocity field prediction in the latent space and the high-frequency residual prediction in the pixel space until the full trajectory denoising iteration is completed and the final target image is output.

[0051] At the end of each diffusion time step, the state update mechanism performs branching based on the aforementioned noise stage. If the current diffusion time step is greater than or equal to a preset time gating threshold, i.e., in the high-noise structure generation stage, the system only initiates latent space state stepping. At this time, based on the latent space prediction results output by the current step, the system uses a flow-matched ordinary differential equation solver (such as the Euler forward integral operator or a higher-order numerical integration stepping operator) to update the current latent variables for the next step along the predicted velocity field direction. This process propagates independently within the latent space manifold to accelerate the convergence of the macroscopic semantic skeleton.

[0052] When the sampling iteration first enters the low-noise detail recovery stage and subsequently, the system initiates a cross-space multi-stream collaborative output and state update mechanism. At the data stream output end, the system maintains dual-end joint control. The latent space prediction head first estimates clean latent variables based on the current flow field velocity and injects them into the decoder of the pre-trained variational autoencoder, mapping the output to the corresponding structural basis image. Simultaneously, the high-frequency pixel prediction head of the pixel space branch outputs the high-frequency residual prediction result corresponding to the current step. After obtaining the visual representations of these two different spaces, the system performs element-wise summation and fusion of the structural basis image output by the decoder and the high-frequency residual prediction result output by the high-frequency pixel prediction head. This physical process of spatially complementary stacking of low-frequency skeleton and high-frequency micro signals is explicitly calculated using the following output-gated fusion formula: ; In the above output gating fusion formula, This represents the intermediate target image obtained by fusion at the current diffusion time step; This represents the clean latent variables estimated by the latent variable predictor head; The decoder space restoration function represents the variational autoencoder. That is, the structural basis image generated by decoding; This represents a time-gated indicator function operator based on time steps; Indicates the current diffusion time step; This indicates the preset time-gating threshold. This represents the high-frequency residual prediction result output by the high-frequency pixel prediction head, used to compensate for details in the spatial compression defects of the decoder. Since we are currently in a low-noise phase, the indicator function is activated, and the high-frequency residual prediction result... Element-wise weighted summation is performed with the structural base image to generate the fused result.

[0053] After generating the fusion result of the current step, the system combines this fusion result with the latent space prediction result to synchronously update the current latent variables for the next step. Specifically, the fused intermediate target image is mapped back to the latent space through the forward path, or it is directly weighted with the latent space prediction result of the current step to correct the drift of the latent space denoising trajectory on the micro-texture. This ensures that the updated current latent variables for the next step have both global semantic consistency and contain high-frequency pixel-level priors. The above denoising iteration process is executed repeatedly in the time domain until the last diffusion time step is completed. The system finally outputs a target image with a stable structure and extremely high texture fidelity.

[0054] In actual system operation and application, the dynamic updates in the above two stages play a crucial role in guiding loss constraints during the joint optimization phase of model training. To enable the network to adaptively learn this staged update strategy, model parameter fine-tuning is based on the joint driving force of flow matching loss and pixel loss. The optimization of the latent space branch is maintained at the full diffusion time step, and its corresponding latent space velocity field flow matching loss is calculated using the following flow matching loss function formula: ; In the above formula for the flow matching loss function, This represents the flow matching loss in the latent space velocity field; Represents the expectation operator; Indicates the diffusion time step Noisy latent variables constructed below; Represents noiseless, real-world latent variables; This represents the standard Gaussian random noise of the sample; This indicates the conditions for obtaining the information; This represents the velocity field vector predicted by the latent space branch. When the time step is less than a preset time gating threshold, to constrain the accuracy of the aforementioned fusion results, the system additionally calculates pixel-level loss through the mean squared error operator and the pre-trained perceptual feature extraction network. Its internal combined structure is expressed by the following pixel-level loss function formula: ; Under the above pixel-level loss function formula, the task... This represents the pixel-level loss at the output of the pixel space branch; Represents the target's true ground-truth image; This represents the intermediate target image obtained by the output gated fusion formula mentioned above; This represents the L2 norm squared mean square error operator used to calculate reconstruction loss; This represents the perceptual loss calculation function used to constrain the consistency of high-level visual features; and These represent the corresponding reconstruction loss weight coefficients and perception loss weight coefficients, respectively. The system ultimately combines the latent space flow matching loss with the pixel-level loss, constructing an overall gradient backpropagation baseline using the following total loss function formula, and jointly optimizing and fine-tuning the parameters of the entire diffusion transformer backbone and prediction head: ; In the above formula for the total loss function, This represents the overall optimization loss that guides network updates. This operating mode establishes a deterministic decoupling between full-time stream matching and subsequent pixel regression through a temporal indicator function. This ensures that high-dimensional pixel targets do not interfere with the convergence of the semantic skeleton during early training, while in the later stages, it forces the high-frequency pixel prediction head to perform continuous and sharp texture micro-compensation on the final target image along the latent space trajectory.

[0055] Furthermore, the internal network parameters of the shared transformer backbone, latent space branch, and pixel space branch are determined through the following joint optimization training process: At each full diffusion time step, the latent space velocity field flow matching loss corresponding to the latent space branch is calculated. When the current diffusion time step is less than the preset time gating threshold, the reconstruction loss and the perception loss are calculated by the mean square error operator and the pre-trained perceptual loss network respectively, and the pixel-level loss is obtained by weighted summation. The latent space velocity field flow matching loss and pixel-level loss are jointly constructed to form a total loss function, and the internal network parameters of the model are jointly optimized and fine-tuned based on the gradient of the total loss function through backpropagation.

[0056] Furthermore, the sampling inference process for performing the denoised trajectory includes the following steps: During the high-noise structure generation stage after the start of denoising sampling, only the latent space branch is activated for iterative denoising; When the denoising sampling iteration first enters the low-noise detail recovery stage, the decoder of the variational autoencoder is called to spatially restore the current latent variables and generate the initial pixel input of the pixel spatial input image. In each subsequent low-noise denoising sampling step, the current latent variable updated in the current step is injected together with the pixel space input image into the shared transformer backbone, so that the output high-frequency residual prediction result is synchronously refined along the latent space trajectory.

[0057] Specifically, in the determination of model parameters and actual deployment applications, the system achieves a complete engineering closed loop through two mutually cooperating forward computation processes: first, network parameters are determined through training the target loss function in a two-stage decoupled manner; second, the flow field is synchronously refined across space during sampling inference using a specific time routing mechanism.

[0058] To enable the shared converter backbone, latent space branch, and pixel space branch to collaboratively model, the determination of its internal network parameters relies on a joint optimization training process that decouples structure and details. This training process maintains basic semantic supervision across the entire diffusion time step. Specifically, for any diffusion time step randomly sampled from a continuous interval, the system utilizes the latent space branch to calculate its corresponding latent space velocity field flow matching loss. This process forces the shared converter backbone to maintain stable global semantic and macroscopic topology modeling capabilities across all time steps by explicitly regressing the linear interpolation vector field between Gaussian noise and the true latent variables. Since the flow matching loss function formula has already been explicitly disclosed above, it will not be repeated here.

[0059] Meanwhile, when the current diffusion time step is less than the preset time gating threshold, the system determines that the denoising trajectory has entered a low-noise period and additionally activates the pixel space branch loss. The system first calculates the L2 squared difference between the output image and the real image using the mean squared error operator to construct the reconstruction loss. Then, the system extracts deep, multi-level visual features through a pre-trained perceptual loss network and calculates the perceptual loss using a distance metric in the feature space to constrain the consistency of high-level visual features. The system then performs a weighted sum of the calculated reconstruction loss and the perceptual loss to obtain the final pixel-level loss. Since the complete explicit derivation of the pixel-level loss function formula and the final summation of the total loss function formula has already been performed, it will not be repeated here. The network performs gradient backpropagation based on the gradient of the total loss function, and uses the optimizer to jointly fine-tune and determine all internal network parameters, including the shared transformer backbone.

[0060] After the network parameters are determined, the system employs a time-gated routing strategy highly consistent with the training process when actually deploying and executing the sampling inference of the denoising trajectory. During the high-noise structure generation phase immediately after the start of denoising sampling, the system controls the underlying computational pipeline to only activate the latent space branch for iterative denoising. Because the diffusion time step in this phase is not less than the preset time-gated threshold, pixel branches remain closed, allowing the network to quickly complete the planning and shaping of global image semantics, background layout, and large-scale subject contours with low sequence dimensions and computational overhead.

[0061] As the sampling time steps decrease, when the denoising sampling iteration first steps to a critical point below the preset time gating threshold—that is, when it first enters the low-noise detail recovery stage—the system calls the decoder of the pre-trained variational autoencoder to spatially restore the current latent variables, recovering the corresponding color image, and using it as the initial pixel input for the pixel space input image. In each subsequent low-noise denoising sampling step, the system no longer initializes the pixel stream from zero. Instead, it uses the current latent variables updated in the current sampling step and the pixel space input image maintained or updated in the previous step as dual-stream feature inputs, uniformly performing block embedding and injecting them into the shared transformer backbone. Within the same transformer architecture, pixel features and latent features interact globally using a self-attention mechanism, ensuring that the high-frequency residual prediction results output by the high-frequency pixel prediction head can always synchronously and progressively refine the texture along the stable semantic trajectory established in the latent space. This inference process ensures that the recovery of pixel-level high-frequency details is entirely based on a reliable semantic skeleton, avoiding training-inference mismatch and the accumulation of artifacts, thus achieving high-quality, high-fidelity image synthesis within a unified framework.

[0062] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating images using a diffusion transformer that combines latent space and pixel space, characterized in that, include: S1. Obtain conditional information and initialize random noise as the current latent variable in the latent space; S2. In each diffusion time step of the denoising trajectory, determine whether the current denoising trajectory is in the high noise structure generation stage or the low noise detail recovery stage based on the noise level at the current diffusion time step. S3. When in the high-noise structure generation stage, the current latent variables and condition information are input into the shared converter backbone, and the latent space prediction result corresponding to the current diffusion time step is output through the latent space branch. S4. When in the low-noise detail recovery stage, the current latent variable is restored through the decoder space of the variational autoencoder, and the corresponding pixel space input image is output. S5. Embed the pixel space input image and the current latent variable features respectively, and inject them together into the shared transformer backbone during the low noise detail recovery stage to perform dual-stream joint feature extraction, and output the latent space prediction result and the high-frequency residual prediction result respectively. S6. When in the high-noise structure generation stage, update the current latent variables in the next latent space according to the latent space prediction results; When in the low-noise detail recovery stage, the pixel space input image is summed and fused with the high-frequency residual prediction result, and the current latent variable in the latent space is updated in the next step by combining the fusion result with the latent space prediction result until the denoising iteration is completed and the target image is output.

2. The image generation method using a diffusion transformer that combines latent space and pixel space according to claim 1, characterized in that, In S1, the condition information to be obtained includes: The input text description is mapped to a conditional token sequence by a text encoder to obtain at least one of the input category label embedding vector or the input visual control signal. The visual control signal includes a visual control token obtained by feature encoding of edge signals, depth signals, or layout signals.

3. The image generation method using a diffusion transformer that combines latent space and pixel space according to claim 1, characterized in that, In S2, determining whether the current denoising trajectory is in the high-noise structure generation stage or the low-noise detail recovery stage includes: Get the current diffusion time step and the preset time gating threshold; Determine whether the current diffusion time step is less than the preset time gating threshold; If the current diffusion time step is greater than or equal to the preset time gating threshold, then the current denoising trajectory is determined to be in the high-noise structure generation stage. If the current diffusion time step is less than the preset time gating threshold, the current denoising trajectory is determined to be in the low noise detail recovery stage.

4. The image generation method using a diffusion transformer that combines latent space and pixel space according to claim 1, characterized in that, In step S4, the spatial downsampling rate used by the decoder of the variational autoencoder to restore the current latent variable is consistent with the image block size used in step S5 when embedding features into the pixel space input image, so that the length of the pixel embedding sequence obtained by the division is the same as the length of the latent variable embedding sequence after the current latent variable space is expanded, and a corresponding arrangement relationship is established between the latent space position and the pixel space position.

5. The image generation method using a diffusion transformer that combines latent space and pixel space according to claim 1, characterized in that, In S5, the joint injection of the shared converter backbone for dual-stream joint feature extraction during the low-noise detail recovery stage includes: In the low-noise detail recovery stage, a temporal gating injection mechanism is used to update the latent variable embedding sequence and the pixel embedding sequence obtained after feature embedding by performing feature gating superposition of the spatial hidden layer to obtain the fused feature sequence. The fused feature sequence is injected into the input of the shared transformer backbone, and dual-stream joint feature extraction is performed within the same transformer architecture.

6. The image generation method using a diffusion transformer that combines latent space and pixel space according to claim 1, characterized in that, In S5, the output high-frequency residual prediction results include: The corresponding hidden states output by the shared transformer backbone are projected into spatial dimensions and rearranged into a two-dimensional feature grid. The two-dimensional feature grid is injected into the high-frequency pixel prediction head, and multiple cascaded convolutional thinning modules are used to progressively amplify and restore the spatial resolution, outputting the high-frequency residual prediction result.

7. The image generation method using a diffusion transformer that combines latent space and pixel space according to claim 6, characterized in that, The stepwise scaling up and restoration of spatial resolution using multiple cascaded convolutional thinning modules includes: The two-dimensional feature grid is controlled to pass through the convolutional layer, sub-pixel convolutional upsampling layer, group normalization layer and activation function layer inside the convolutional thinning module in turn, and the feature grid is enlarged step by step by utilizing local feature integration and pixel rearrangement operations.

8. The image generation method using a diffusion transformer that combines latent space and pixel space according to claim 6, characterized in that, The process of outputting high-frequency residual prediction results from the high-frequency pixel prediction head also includes a conditional modulation procedure: The global condition vector is obtained by summing the time step embeddings and the condition embeddings corresponding to the condition information. A multilayer perceptron is used to linearly map the global conditional vector to generate corresponding scaling and translation parameters. The scaling and translation parameters are used to perform adaptive layer normalization control on the feature mesh in the progressive magnification and restoration process.

9. The image generation method using a diffusion transformer that combines latent space and pixel space according to claim 1, characterized in that, The internal network parameters of the shared transformer backbone, latent space branch, and pixel space branch are determined through the following joint optimization training process: At each full diffusion time step, the latent space velocity field flow matching loss corresponding to the latent space branch is calculated. When the current diffusion time step is less than the preset time gating threshold, the reconstruction loss and the perception loss are calculated by the mean square error operator and the pre-trained perceptual loss network respectively, and the pixel-level loss is obtained by weighted summation. The latent space velocity field flow matching loss and pixel-level loss are jointly constructed to form a total loss function, and the internal network parameters of the model are jointly optimized and fine-tuned based on the gradient of the total loss function through backpropagation.

10. The image generation method using a diffusion transformer that combines latent space and pixel space according to claim 1, characterized in that, The sampling inference process for performing denoised trajectories includes the following steps: During the high-noise structure generation stage after the start of denoising sampling, only the latent space branch is activated for iterative denoising; When the denoising sampling iteration first enters the low-noise detail recovery stage, the decoder of the variational autoencoder is called to spatially restore the current latent variables and generate the initial pixel input of the pixel spatial input image. In each subsequent low-noise denoising sampling step, the current latent variable updated in the current step is injected together with the pixel space input image into the shared transformer backbone, so that the output high-frequency residual prediction result is synchronously refined along the latent space trajectory.

Images