Semi-autoregressive hierarchical text generation method based on discrete flow matching diffusion algorithm

Abstract

The invention discloses a semi-autoregressive hierarchical text generation method based on a discrete flow matching diffusion algorithm, which comprises the following steps of: (1) constructing a hierarchical block diffusion language model, firstly dividing a text sequence into large blocks by the model, generating in an autoregressive manner among the blocks, and generating in parallel in the blocks in a discrete flow matching diffusion manner; (2) training the hierarchical block diffusion language model to enable the hierarchical block diffusion language model to learn conditional distribution between blocks and joint distribution in the blocks; (3) recursively decomposing each large block into smaller sub-blocks by adopting a hierarchical block structure to form a multi-layer block tree structure; and (4) in a reasoning stage, adopting a top-down generation mode, firstly generating an upper-layer large block, and then generating a lower-layer sub-block based on a generated content condition, thereby realizing efficient text generation. By means of the method and device, the text generation efficiency can be remarkably improved while the language modeling quality is kept.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and in particular relates to a semi-autoregressive hierarchical text generation method based on a discrete flow matching diffusion algorithm. Background Art

[0002] With the development of artificial intelligence technology, the field of natural language processing has made significant progress, and large language models have demonstrated powerful capabilities in text generation. Currently, there are two main paradigms for natural language generation: autoregressive models and diffusion models.

[0003] The autoregressive model is the current mainstream language model architecture. It generates each character unit (token) in a sequence from left to right, and the prediction at each position depends on all previously generated character units. The autoregressive model has high generation quality, but due to its inherent sequential generation nature, it cannot use parallel computing to accelerate the generation process, resulting in low efficiency in long text generation scenarios. In practical applications, the autoregressive model requires L steps to generate a text sequence of length L, and the generation speed is proportional to the sequence length. This becomes a major bottleneck in application scenarios that require real-time response.

[0004] To address the efficiency issues of the autoregressive model, researchers proposed a diffusion-based language model. The diffusion model gradually adds noise to the data by defining a forward process, and then learns a reverse denoising process to generate data. Unlike the autoregressive model, the diffusion model can generate character units in parallel, which can theoretically significantly improve generation efficiency. However, the standard diffusion language model also faces some challenges, including fixed-length generation limitations, the inability to effectively utilize the KV cache mechanism, and a large gap compared to the autoregressive model in standard evaluation metrics such as perplexity.

[0005] Recent research has proposed semi-autoregressive Block Diffusion Language Models (BD-LMs). This method combines the advantages of autoregressive and diffusion models by decomposing text sequences into multiple blocks, generating them using an autoregressive approach between blocks, and using a diffusion model to generate multiple character units in parallel within a block. This hybrid approach retains the flexibility of the autoregressive model while partially utilizing the parallel generation capability of the diffusion model. Experiments show that the block diffusion model can generate sequences of arbitrary length and outperforms the pure diffusion model in terms of perplexity metrics. However, as the block size increases, the perplexity increases accordingly, indicating a trade-off between block size and generation quality.

[0006] On the other hand, Discrete Flow Matching (DFM), an emerging generative paradigm, connects the source and target distributions by constructing a continuous-time Markov chain model and uses a simpler training objective to learn the data distribution. Compared to traditional diffusion models, DFM offers potential advantages in training stability and sampling efficiency, but it has not yet been widely used in large-scale language models.

[0007] Furthermore, most existing block diffusion methods use fixed-size blocks for generation. While this approach is simple and straightforward, it fails to fully exploit the hierarchical relationships between different semantic segments. Research has shown that varying the block size from small to large significantly varies model performance and efficiency. For example, increasing the block size from 4 to 128 results in only a slight increase in perplexity, but can improve generation efficiency by dozens of times. This suggests that applying parallel generation to larger semantic units and then further refining them at a finer granularity may be an effective strategy.

[0008] In summary, existing technologies for language model generation still face a trade-off between efficiency and quality. Combining the advantages of discrete stream matching and block diffusion, while simultaneously introducing a hierarchical structure to balance generation efficiency and quality, presents a current technical challenge. Therefore, developing a novel language model generation method that significantly improves generation efficiency while maintaining quality holds significant research value and promising applications. Summary of the Invention

[0009] The present invention provides a semi-autoregressive hierarchical text generation method based on a discrete flow matching diffusion algorithm, which can significantly improve the text generation efficiency while maintaining the quality of language modeling.

[0010] A semi-autoregressive hierarchical text generation method based on a discrete flow matching diffusion algorithm includes the following steps:

[0011] (1) A hierarchical block diffusion language model is constructed. The model first divides the text sequence into large blocks, generates between blocks using autoregressive method, and generates within blocks using discrete flow matching diffusion method in parallel;

[0012] (2) Training the hierarchical block diffusion language model to learn the conditional distribution between blocks and the joint distribution within blocks;

[0013] (3) Using a hierarchical block structure, each large block is recursively decomposed into smaller sub-blocks to form a multi-level block tree structure;

[0014] (4) In the inference stage, a top-down generation approach is adopted, where the upper-level large blocks are first generated, and then the lower-level sub-blocks are generated based on the generated content conditions, thereby achieving efficient text generation.

[0015] The method proposed in this invention first decomposes a long text sequence into multiple large blocks. The blocks are generated in an autoregressive manner, that is, one block is generated at a time. Within each block, a diffusion model with discrete flow matching is used for parallel generation, rather than the traditional discrete denoising diffusion method. This hybrid architecture retains the powerful modeling capabilities of the autoregressive model while leveraging the parallel generation advantages of the diffusion model. By introducing a hierarchical block structure, the model can adaptively adjust the degree of parallelism to achieve a better balance between text quality and generation speed. This method can effectively improve the efficiency of long text generation while maintaining a low perplexity, and is suitable for a variety of application scenarios requiring efficient text generation.

[0016] In step (1), the hierarchical block diffusion language model divides the text sequence of length L into B large blocks, each of which is of length L′, such that B = L / L′; for the relationship between blocks, an autoregressive approach is used to model the relationship:

[0017]

[0018] Where p is the probability density function, x b represents the bth block, x ＜b Represents all blocks before the b-th block.

[0019] In step (1), for the joint distribution p(x b |x ＜b ), different from the discrete denoising diffusion model used in the traditional block diffusion method, the present invention adopts the discrete flow matching method to build the model. Discrete flow matching constructs a probabilistic path from the starting distribution to the target distribution and learns the velocity field that generates the path, thereby achieving an efficient generation process. Specifically, for the intra-block distribution p(x b |x ＜b ), the discrete flow matching is defined as follows:

[0020] (1-1) Define the probability path p t (x b |x ＜b ), so that p0(x b |x ＜b )=p0(x b ) is a simple prior distribution, p1(x b |x ＜b )=p(x b |x ＜b ) is the target conditional distribution;

[0021] (1-2) Define the continuous-time Markov chain model, where the velocity field u t (y,x|x ＜b ) represents the rate of transition from state x to state y;

[0022] (1-3) Training velocity field u t It generates the defined probabilistic path by minimizing the discrete flow matching loss function:

[0023]

[0024] Among them, E represents the mathematical expectation, t represents the tth step, X t represents the state at step t, represents the contribution of the Bregman divergence on a single component, and S represents the state space, which is used to measure the difference between the predicted velocity field and the true velocity field.

[0025] Furthermore, the velocity field is calculated by decomposing the velocity field. The formula is:

[0026]

[0027] in, represents all positions except the i-th position, and δ is the Kronecker function. This decomposition method will originally require V L′ The velocity field of dimension L′·V (V is the vocabulary size) is simplified to L′·V dimension, which greatly reduces the computational complexity.

[0028] Furthermore, the generalized KL divergence is chosen as the specific form of the Bregman divergence.

[0029] In step (2), the hierarchical block diffusion language model is trained. The specific process is as follows:

[0030] (2-1) Using a large-scale corpus as training data, the text sequence is segmented into blocks of a predetermined size L′;

[0031] (2-2) For the inter-block autoregressive part, maximum likelihood is used for training;

[0032] (2-3) For the discrete flow matching part within the block, the discrete flow matching loss function is used for training;

[0033] (2-4) Combine the two loss functions to form a joint training objective:

[0034]

[0035] Among them, λ is a hyperparameter that balances the two parts of loss, is the discrete flow matching loss of the b-th block.

[0036] In step (3), the construction process of the hierarchical block structure is:

[0037] (3-1) Divide the text sequence into large blocks of size L′1 to form the first-level block structure;

[0038] (3-2) Recursively divide each large block into two parts to form a second layer of sub-blocks of size L′2;

[0039] (3-3) Continue recursively decomposing to form multi-layer sub-block structures such as L′3 and L′4;

[0040] (3-4) Construct hierarchical conditional relationships, that is, each sub-block not only depends on all previous large blocks, but also specifically depends on the first half of the content of the upper-level block to which it belongs.

[0041] In the hierarchical block structure, the block sizes of each layer are:

[0042] The first layer block size L′1=128; the second layer block size L′2=64; the third layer block size L′3=32; the fourth layer block size L′4=16; the fifth layer block size L′5=8; and the sixth layer block size L′6=4.

[0043] The specific process of step (4) is:

[0044] (4-1) First, autoregressively generate the first layer of large blocks;

[0045] (4-2) For each generated large block, the intra-block discrete flow matching diffusion process is performed in parallel, but instead of directly generating the final text, the block representation of the next layer is generated;

[0046] (4-3) Recursively perform similar operations on each layer until the bottom layer;

[0047] (4-4) At the bottom layer, the discrete flow matching diffusion method is used to generate the final character unit sequence in parallel.

[0048] In step (4-4), the discrete flow matching diffusion method is adopted, and the specific process is as follows:

[0049] Initialization: For each position in the block, the initial state is set to the mask mark;

[0050] Sampling process: Starting from t = 1, the t value is gradually reduced until t = 0. At each step, the state is updated by the discrete flow matching sampling function according to the current t value;

[0051] Sampling function implementation: In the hybrid path framework, each sampling step includes predicting the posterior probability sampling And based on the sampling and the current state x i Calculate transition probabilities and perform state updates.

[0052] This hierarchical generation strategy significantly improves generation efficiency while maintaining low perplexity by combining different levels of parallelism. In particular, by using blocks of varying sizes at different levels, the model can adaptively adjust the degree of parallelism during generation, using larger blocks for parallel generation of easily predictable parts and smaller blocks for more refined modeling of less predictable parts.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention applies the discrete stream matching method to the block diffusion language model. Compared with the traditional discrete denoising diffusion method, discrete stream matching can provide more accurate likelihood estimation and requires fewer sampling steps, thereby improving generation efficiency.

[0055] 2. The present invention adopts a hierarchical block structure, which can better capture the long-range dependencies of text by recursively decomposing large blocks into small blocks and establishing conditional dependencies between different levels while maintaining high generation efficiency.

[0056] 3. The present invention supports a flexible pre-generation strategy. When computing resources are limited, it can give priority to generating upper-level large blocks, provide users with a quick preview, and then gradually refine them into the final high-quality text, thereby improving the user experience.

[0057] 4. The present invention can be seamlessly integrated with the existing large language model architecture. Only the decoder part needs to be modified, and there is no need to retrain the entire model. It has good practicality and compatibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is an overall flow chart of a semi-autoregressive hierarchical text generation method based on a discrete flow matching diffusion algorithm according to the present invention.

[0059] Figure 2 This is a schematic diagram of the hierarchical block structure in the present invention.

[0060] Figure 3 This is a flow chart of the generation mechanism within the discrete flow matching block in the present invention.

[0061] Figure 4 This is a specific flow chart of the training or reasoning process of the present invention.

[0062] Figure 5 A schematic diagram of the model structure. DETAILED DESCRIPTION

[0063] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0064] like Figure 1 As shown, a semi-autoregressive hierarchical text generation method based on discrete flow matching diffusion algorithm includes the following steps:

[0065] Step S1: Construct a hierarchical block diffusion language model framework. In the actual implementation process, the model architecture is based on the DiT model.

[0066] Step S2: Implementing the intra-block generation mechanism of discrete flow matching.

[0067] For each block distribution, the present invention adopts the discrete flow matching method defined as follows. The specific process is as follows: Figure 3 As shown:

[0068] (S2-1) define probability paths;

[0069] (S2-2) Construct a continuous time Markov chain (CTMC) model;

[0070] (S2-3) Training speed field u t So that it generates the defined probability path. In actual implementation, the generalized KL divergence is chosen as the specific form of Bregman divergence:

[0071]

[0072] where u j 、v j Represent the jth component of vectors u and v respectively. In order to achieve efficient velocity field calculation, the present invention adopts the method of decomposing the velocity field, namely:

[0073]

[0074] in, represents all positions except the i-th position, and δ is the Kronecker function. This decomposition method will originally require V L′ The velocity field of dimension L′·V (V is the vocabulary size) is simplified to L′·V dimension, which greatly reduces the computational complexity.

[0075] Step S3, construct a hierarchical block structure, such as Figure 2 shown.

[0076] The present invention organizes the text sequence into a multi-level block structure, which is specifically implemented as follows:

[0077] (S3-1) First layer: Divide the text sequence into large blocks of size L′1=128;

[0078] (S3-2) Second layer: Recursively divide each large block into two parts to form second-layer sub-blocks of size L′2=64;

[0079] (S3-3) Continue recursively decomposing to form a multi-layer sub-block structure with L′3=32, L′4=16, L′5=8, and L′6=4;

[0080] (S3-4) Construct hierarchical conditional relations: Each sub-block depends not only on all previous large blocks, but also on the first half of the content of the upper block to which it belongs, e.g. Figure 4 shown.

[0081] For example, consider a text sequence of length 512, which is first divided into 4 large blocks of length 128. The first large block is divided into two sub-blocks of length 64, and so on until the minimum block size is 4. This forms a 6-level block tree structure, with each level containing blocks of block sizes of 128, 64, 32, 16, 8, and 4, respectively.

[0082] The hierarchical block structure enhances the model's understanding of context through a special conditional dependency design. For example, in layer n, the i-th child block not only relies on the traditional autoregressive dependency of all previous blocks, but also specifically relies on the first half of the corresponding parent block in layer n-1. This design enables the model to better capture local contextual information, improving generation quality.

[0083] Step S4, model training process.

[0084] Model training is performed using the following steps:

[0085] (S4-1) Data preparation: Use a large-scale corpus (such as C4, OpenWebText, etc.) as training data and segment the text sequence into a predetermined block size L′;

[0086] (S4-2) Model initialization: Initialize parameters from a pre-trained language model (such as BERT, GPT series), or randomly initialize;

[0087] (S4-3) Training objective construction: Using the joint training objective mentioned above, for the inter-block autoregressive part, a cross entropy loss is used; for the intra-block discrete flow matching part, a discrete flow matching loss function is used. In this embodiment, λ is set to 0.5;

[0088] (S4-4) Training process: Adam optimizer is used, the initial learning rate is set to 5e-5, the cosine decay scheduling strategy is used, and the batch size is 256;

[0089] (S4-5) Hierarchical training strategy: First train on a smaller block size (such as L′=4 or L′=8), then gradually increase the block size, and finally fine-tune on the complete hierarchical structure.

[0090] Step S5, text generation in the reasoning stage, the model structure is as follows Figure 5 shown.

[0091] In the reasoning stage, the present invention adopts a top-down hierarchical generation strategy to achieve efficient text generation:

[0092] (S5-1) First, the first layer of large blocks (L′1=128) are generated autoregressively, that is, one large block is generated at a time, and then the generated large block is used as a condition to generate the next large block;

[0093] (S5-2) For each generated large block, the discrete flow matching diffusion process within the block is performed in parallel. The specific steps are as follows:

[0094] a. Initialization: For each position in the block, the initial state is set to the mask mark;

[0095] b. Sampling process: Starting from t = 1, the t value is gradually reduced until t = 0. At each step, the state is updated by the discrete flow matching sampling function according to the current t value;

[0096] c. Sampling function implementation: In the hybrid path framework, each sampling step is implemented as follows:

[0097] Predicted posterior probability sampling According to the sampling and the current state x i , calculate transition probabilities and perform state updates.

[0098] (S5-3) Generate the next layer: Using the content of the first layer block as a condition, generate the sub-blocks of the second layer in parallel;

[0099] (S5-4) Recursive generation: Continue to perform similar operations on each layer until the final text generation is completed at the bottom layer (L′6=4).

[0100] This hierarchical generation strategy can flexibly adjust the degree of parallelism at different levels, using larger blocks for parallel generation of easily predictable text regions and smaller blocks for fine modeling of complex regions, thereby improving generation efficiency while maintaining quality.

[0101] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A semi-autoregressive hierarchical text generation method based on discrete flow matching diffusion algorithm, characterized in that: The following steps are involved: (1) A hierarchical block diffusion language model is constructed. The model first divides the text sequence into large blocks, generates between blocks using autoregressive method, and generates within blocks using discrete flow matching diffusion method in parallel; (2) Training the hierarchical block diffusion language model to learn the conditional distribution between blocks and the joint distribution within blocks; (3) Using a hierarchical block structure, each large block is recursively decomposed into smaller sub-blocks to form a multi-level block tree structure; (4) In the inference stage, a top-down generation approach is adopted, where the upper-level large blocks are first generated, and then the lower-level sub-blocks are generated based on the generated content conditions, thereby achieving efficient text generation.

2. The semi-autoregressive hierarchical text generation method based on the discrete flow matching diffusion algorithm according to claim 1 is characterized in that: In step (1), the hierarchical block diffusion language model divides the text sequence of length L into B large blocks, each of which is L in length. ′ , so that B = L / L ′ ; For the relationship between blocks, autoregressive modeling is used: Where p is the probability density function, x b represents the bth block, x ＜b Represents all blocks before the b-th block.

3. The semi-autoregressive hierarchical text generation method based on discrete flow matching diffusion algorithm according to claim 2 is characterized in that: In step (1), the block is generated in parallel using the discrete flow matching diffusion method. For the distribution p(x b |x ＜b ), the discrete flow matching is defined as follows: (1-1) Define the probability path p t (x b |x ＜b ), so that p0(x b |x ＜b )=p0(x b ) is a simple prior distribution, p1(x b |x ＜b )=p(x b |x ＜b ) is the target conditional distribution; (1-2) Define the continuous-time Markov chain model, where the velocity field u t (y,x|x ＜b ) represents the rate of transition from state x to state y; (1-3) Training velocity field u t It generates the defined probabilistic path by minimizing the discrete flow matching loss function: Among them, E represents the mathematical expectation, t represents the tth step, X t represents the state at step t, represents the contribution of the Bregman divergence on a single component, and S represents the state space, which is used to measure the difference between the predicted velocity field and the true velocity field.

4. The semi-autoregressive hierarchical text generation method based on discrete flow matching diffusion algorithm according to claim 3 is characterized in that: The velocity field is calculated by decomposing the velocity field. The formula is: in, represents all positions except the i-th position, and δ is the Kronecker function.

5. The semi-autoregressive hierarchical text generation method based on discrete flow matching diffusion algorithm according to claim 3 is characterized in that: The generalized KL divergence is chosen as the specific form of Bregman divergence.

6. The semi-autoregressive hierarchical text generation method based on discrete flow matching diffusion algorithm according to claim 3 is characterized in that: In step (2), the hierarchical block diffusion language model is trained. The specific process is as follows: (2-1) Using a large-scale corpus as training data, the text sequence is divided into blocks of a predetermined size L. ′ Perform segmentation; (2-2) For the inter-block autoregressive part, maximum likelihood is used for training; (2-3) For the discrete flow matching part within the block, the discrete flow matching loss function is used for training; (2-4) Combine the two loss functions to form a joint training objective: Among them, λ is a hyperparameter that balances the two parts of loss, is the discrete flow matching loss of the b-th block.

7. The semi-autoregressive hierarchical text generation method based on discrete flow matching diffusion algorithm according to claim 1 is characterized in that: In step (3), the construction process of the hierarchical block structure is: (3-1) Divide the text sequence into L ′ 1 large blocks, forming the first layer of block structure; (3-2) Recursively divide each large block into two parts to form a block of size L ′ 2 second-layer sub-block; (3-3) Continue recursive decomposition to form a multi-layer sub-block structure; (3-4) Construct hierarchical conditional relationships, that is, each sub-block not only depends on all previous large blocks, but also specifically depends on the first half of the content of the upper-level block to which it belongs.

8. The semi-autoregressive hierarchical text generation method based on discrete flow matching diffusion algorithm according to claim 7 is characterized in that: In the hierarchical block structure, the block sizes of each layer are: The first layer block size L ′ 1=128; second layer block size L ′ 2=64; third layer block size L ′ 3=32; fourth layer block size L ′ 4=16; fifth layer block size L ′ 5=8; sixth layer block size L ′ 6=4.

9. The semi-autoregressive hierarchical text generation method based on discrete flow matching diffusion algorithm according to claim 1 is characterized in that: The specific process of step (4) is: (4-1) First, autoregressively generate the first layer of large blocks; (4-2) For each generated large block, the intra-block discrete flow matching diffusion process is performed in parallel, but instead of directly generating the final text, the block representation of the next layer is generated; (4-3) Recursively perform similar operations on each layer until the bottom layer; (4-4) At the bottom layer, the discrete flow matching diffusion method is used to generate the final character unit sequence in parallel.

10. The semi-autoregressive hierarchical text generation method based on discrete flow matching diffusion algorithm according to claim 9 is characterized in that: In step (4-4), the discrete flow matching diffusion method is adopted, and the specific process is as follows: Initialization: For each position in the block, the initial state is set to the mask mark; Sampling process: Starting from t = 1, the t value is gradually reduced until t = 0. At each step, the state is updated by the discrete flow matching sampling function according to the current t value; Sampling function implementation: In the hybrid path framework, each sampling step includes predicting the posterior probability sampling And based on the sampling and the current state x i Calculate transition probabilities and perform state updates.

Images