A block diffusion approximate linear attention calculation method and model training device for a gated incremental network base model

Abstract

The application discloses a block diffusion approximate linear attention calculation method and a model training device for a gated incremental network basic model. After obtaining the query, the key, the value, the gating coefficient and the decay parameter corresponding to the gated incremental network layer, the causal recursive state update of the gated incremental network layer is approximately expanded into an explicit pairwise aggregation in a limited sliding window. A block diffusion visibility control item is embedded in the explicit pairwise aggregation, an approximate kernel in a unified calculation form is constructed, and then based on the block diffusion visibility control item, the causal sliding window aggregation is performed on the clean label, the clean prefix aggregation and the bidirectional local aggregation in the current noise block are performed on the noise label respectively. Finally, the outputs of various paths are merged, and the output result remains compatible with the interface of the gated incremental network layer. The method can control the block diffusion visibility and prevent the training target from leaking while retaining the original projection, gating and decay structure.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular to a block diffusion approximate linear attention calculation method and model training device for gated incremental network basic models. Background Technology

[0002] Autoregressive large language models typically predict the next label from left to right. Standard Transformer self-attention can flexibly express various visibility relationships through masks, but the computational and storage complexity of global attention increases quadratically with sequence length, making it costly in long context training.

[0003] To reduce complexity, linear attention, gated linear attention, and state-space models can be used to replace some or all of the global attention layers. Examples include native gated incremental networks (GDN, Gated DeltaNet), block diffusion or SDAR-like language modeling schemes, other recursive gated linear attention, and simple sliding window approximate linear attention. GDN is a representative recursive gated linear attention structure, as described in the paper "Gated Delta Networks: Improving Mamba2 with Delta Rule, arXiv:2412.06464". It combines a gating mechanism with DeltaNet, expressing historical context through recursive states of fixed or approximately fixed size. Native GDN or other recursive gated linear attention compress history through recursive states, making them suitable for strictly causal modeling. Their advantages lie in efficient sequence modeling and reasoning friendliness, but their computational semantics are based on causal recursion, compressing history into a single state. Therefore, it cannot directly express bidirectional visibility within a noised block, nor can it clearly distinguish the history of the clean prefix, the local history of the noised block, and the block boundary between them in the same recursive state. Block diffusion language models (such as "Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models", arXiv:2503.09573) and SDAR-like methods (such as "SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation", arXiv:2510.06303) define training or inference paradigms such as intra-block denoising and inter-block sequence generation. They organize sequences into blocks, perform diffusion-like or denoising modeling within blocks, and maintain autoregressive or sequential relationships between blocks, demonstrating the effectiveness of the block diffusion training paradigm itself. However, they do not solve the problem of how to achieve block diffusion visibility on the GDN base model while preserving its original weight structure and inductive bias. Simply approximating GDN as a regular SW-Approx-Linear is not enough to adapt to SDAR2, because block diffusion visibility, mixed cumulative decay, same-block bidirectional paths, and local convolutional patching consistent with this visibility are also required.If the base model natively uses ordinary sliding window attention, then only the visibility mask needs to be modified to adapt to block diffusion, and then complex visibility can be expressed through the mask. However, this is not the native computation form of the GDN base model. Therefore, for the GDN base model, directly replacing it with ordinary sliding window attention will abandon the decay, gating and recursive state induction bias of GDN, which will significantly change the layer structure and parameter usage of the base model, and will not be conducive to low-cost retraining using the original model weights.

[0004] Furthermore, in engineering implementation, if the above path is directly implemented using segmented loops and ordinary tensor operations, a large number of small operators, tensor concatenations, data copies, and synchronization points will be generated, resulting in unstable training throughput and graphics processor utilization. Summary of the Invention

[0005] To address some or all of the problems in existing technologies, and in order to rewrite the recursive historical state update of GDN into an explicit sliding window pairwise aggregation computation while preserving as much of the original projection, gating, attenuation, normalization, and output interface of the GDN basic model as possible, and to introduce block diffusion visibility into this approximation kernel to support clean causal sliding window paths, noised clean-prefix paths, and noised same-block paths, the first aspect of this invention provides a block diffusion approximate linear attention computation method for a gated incremental network basic model, comprising: Obtain the Q, K, V, β gating parameters and attenuation parameters corresponding to the Gated DeltaNet (GDN) layer; The causal recursive state update of the gated incremental network layer is approximately expanded into explicit pairwise aggregation within a finite sliding window; A block diffusion visibility control term is embedded in the explicit pairwise aggregation to construct an approximate kernel with a unified computational form; Based on the block diffusion visibility control term, causal sliding window aggregation is performed on clean tokens, and clean prefix aggregation and bidirectional local aggregation within the current noise block are performed on noise tokens, respectively. The outputs of each path are merged, and the output results are kept compatible with the interface of the gated incremental network layer.

[0006] Furthermore, the approximate expansion of the causal recursive state update of the gated incremental network layer into explicit pairwise aggregation within a finite sliding window includes: Multiply the query vector by After scaling, the vector is then L2 normalized with the key vector, where The dimension of the query vector; The paired effects of query tokens and key tokens are explicitly calculated within the window, while preserving the original projection, gating parameters, and decay parameters of the gated incremental network.

[0007] Furthermore, the explicit calculation of the pairwise operation of query tokens and key tokens within the window includes: The state transition product term is approximated by an exponential decay kernel determined by the cumulative decay: Given a finite window size of W, we define a set of causal sliding windows with the t-th label for clean causal paths and noisy clean prefix sub-paths, and define a bidirectional local window constrained by both block boundaries and positional distances for noisy block sub-paths: , The output of the gated incremental network is then approximated as a pairwise aggregation within an explicit sliding window: , in, For gating or writing coefficients, The cumulative decay is obtained by accumulating the decay parameters for each marker. For the first Attenuation parameters at each location, This represents the relative attenuation term. The label obtained by projecting the gated incremental network layer The query vector, The first layer obtained by projecting the gated incremental network layer A key vector at each position, The first layer obtained by projecting the gated incremental network layer A vector of values ​​at each position.

[0008] Furthermore, the explicit calculation of the pairwise operation of query tokens and key tokens within the window also includes: The relative attenuation term is truncated and expressed as: , in, This is the truncation threshold; Path-specific attenuation kernels are represented as: , in Used for clean causal paths and noisy clean prefix subpaths. Used for noise-prone dual-line paths.

[0009] Furthermore, the block diffusion visibility control term is: wherein, is a Clean Block, is a Noised Block, is a visible set of Clean Query, is a visible set of Noised Query, the visible set of Clean Query is a causal sliding window that does not cross sample boundaries, and the visible set of Noised Query is the union of a clean prefix causal window before the current noised block and a bidirectional local window of the current noised block.

[0010] Further, performing clean prefix aggregation and bidirectional local aggregation within the current noised block on noised tokens respectively comprises: constructing a Clean-Prefix subpath, and aggregating the previous clean prefix before the current noised block as cross-block history; constructing a Same-Block subpath to perform bidirectional local denoising aggregation between tokens within the current noised block, wherein visible positions of the same-block subpath satisfy block(i)=block(t) and |i-t|<W, when W≥B, the same-block subpath covers all positions of the current noised block, wherein B is the size of the noised block; adding the output of the clean prefix subpath and the output of the same-block subpath to obtain a final output of the noised token.

[0011] Further, the clean causal path, the clean prefix subpath and the same-block subpath adopt a path-specific cumulative decay mechanism, wherein the clean causal path and the clean prefix subpath adopt a directed difference between a cumulative decay amount at a query position and a cumulative decay amount at a key position; the same-block subpath adopts the negative value of the absolute value of the difference between the cumulative decay amount at the query position and the cumulative decay amount at the key position, so as to form symmetric exponential decay suitable for bidirectional positions.

[0012] Further, the same-block subpath performs QK similarity calculation, exponential decay calculation, β gating calculation and V aggregation calculation within the current noised block.

[0013] Further, when the gated incremental network layer comprises a causal convolution or a local feature module, a noised input is read for historical positions inside the current noised block, and a corresponding clean prefix input is read for historical positions crossing the starting point of the current noised block.

[0014] Furthermore, merging the outputs of each path includes: placing the sum of the clean prefix sub-paths and the sum of the sub-paths in the same block into the same output buffer, or grouping them into groups of the same segments for continuous scheduling.

[0015] Based on the block diffusion approximate linear attention calculation method described above, a second aspect of the present invention provides a model training apparatus, comprising: The training sample construction unit is used to construct training samples containing noisy segments and corresponding clean segments, and to generate sample boundaries, noise and clean segmentation locations, and block numbers. The gated incremental network layer projection and state expansion unit is used to obtain the Q, K, V, β gate parameters and decay parameters, and to approximately expand the causal recursive state update into explicit pairwise aggregation within a finite sliding window; The visibility control unit is used to construct a clean causal sliding window, a noisy clean prefix window, and a noisy block bidirectional local window, and embeds the corresponding visibility control terms into the explicit pairwise aggregation to construct an approximate kernel with a unified computational form; The path calculation unit is used to perform causal sliding window aggregation on clean tags, clean prefix aggregation and bidirectional local aggregation on noisy tags, and to use path-specific cumulative attenuation; The output merging unit, loss calculation, and parameter update are used to merge the outputs of each path, obtain the model output through an output interface compatible with the gated incremental network layer, calculate the loss according to the block diffusion training objective, and update the model parameters based on the loss.

[0016] This invention provides a block-diffusion approximate linear attention computation method for a gated incremental network (GDN) basic model. It approximates the implicit recursive state update of the GDN layer as explicit pairwise aggregation within a finite sliding window, and sets up a clean causal sliding window, a noisy clean prefix window, and a noisy block bidirectional local window. The corresponding visibility is directly embedded into the entire process of attenuation, gating, and V-aggregation of the approximate kernel. This method preserves the original GDN layer's projection, gating, attenuation, and output interface, ensuring that clean paths maintain their original causal semantics. Simultaneously, it allows noisy paths to perform bidirectional denoising within the same block without reading the current clean block and other noisy blocks, thereby reducing the risk of training target leakage. Attached Figure Description

[0017] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the various embodiments of the present invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0018] Figure 1The diagram illustrates a block diffusion approximate linear attention calculation method for a gated incremental network basic model according to an embodiment of the present invention. Figure 2 This diagram illustrates a block-diffusion approximate linear attention calculation method for a basic model of a gated incremental network according to an embodiment of the present invention. Figure 3 The diagram shows a structural schematic of a model training device according to an embodiment of the present invention. Detailed Implementation

[0019] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or in conjunction with other alternatives and / or additional methods or components. In other instances, well-known structures or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific numbers and configurations are set forth to provide a comprehensive understanding of embodiments of the invention. However, the invention is not limited to these specific details. Furthermore, it should be understood that the embodiments shown in the drawings are illustrative representations and are not necessarily drawn to scale.

[0020] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.

[0021] It should be noted that the embodiments of the present invention describe the method steps in a specific order; however, this is only for illustrating the specific embodiment and not for limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0022] GDN compresses historical context into causal recursive states, making it suitable for time-sequential autoregressive modeling. However, in SDAR2's block diffusion training, noisy labels need to simultaneously read previous clean prefixes and perform local bidirectional interactions within the current noisy block. If the causal recursive state of GDN is directly used, the bidirectional information within the current block and previous clean historical information will be mixed into the same recursive state, making it difficult to accurately control the source of visibility. Based on this, in order to retain as much of the original projection, gating, attenuation, normalization, and output interface of the GDN basic model as possible, the recursive historical state update of GDN is approximately rewritten as an explicit sliding window pairwise aggregation calculation. Block diffusion visibility is introduced into this approximation kernel, enabling it to support clean causal sliding window paths, noised clean-prefix paths, and noised same-block bidirectional paths. This invention provides a block-diffusion approximate linear attention calculation method for basic gated incremental network models, and further addresses the feasibility of the above-mentioned modifications in large model training, including how to construct hybrid cumulative decay, how to make local convolutional features consistent with block diffusion visibility, and how to reduce training computation fragmentation caused by GDN unrolling and block diffusion path splitting through segmented pre-computation, block alignment, group execution, and custom GPU kernel functions.

[0023] The block diffusion approximate linear attention calculation method provided by this invention does not directly invent a block diffusion training paradigm, nor does it add a block diffusion mask to ordinary Transformer attention. Instead, it rewrites the original implicit causal recursive state update into an explicit sliding window local pairwise aggregation within the GDN base model layer, and adds block diffusion visibility control to the aggregation kernel.

[0024] In one embodiment of the present invention, the input projection of the GDN layer still generates intermediate quantities such as query Q, key K, value V, gating coefficient β, and decay parameter g. Before performing pairwise weight calculation, L2 normalization is performed on Q and K, and Q can be scaled according to the square root of the query vector dimension. The original recursive state update is replaced or approximated as pairwise weight calculation within a finite sliding window. in, For block diffusion visibility control item, The relative decay term is determined by the cumulative decay amount. Both the block diffusion visibility control term and the relative decay term are determined separately according to the path. Clean paths are aggregated along a causal sliding window that does not cross sample boundaries; noised paths are decomposed into clean-prefix causal aggregation before the noised block and same-block bidirectional aggregation within the current noised block.

[0025] The aforementioned approximate kernel is further combined with hybrid cumulative attenuation g_abs, where Clean causal paths and noisy clean prefix paths use the directed difference between the query cumulative amount and the key cumulative amount; noisy block paths use the inverse of the absolute value of their difference, forming symmetrical attenuation. Thus, the attenuation and gating structure, which is approximately consistent with GDN, can still be used, while ensuring that historical sources conform to block diffusion semantics.

[0026] If the original GDN layer contains causal depthwise convolutions or similar local feature modules, then the cross-block convolution history of the Noised branch can be further cleaned up with Clean-Prefix to keep local convolutions, cumulative decay, and attention visibility consistent.

[0027] The technical solution of the present invention will be further described below with reference to the accompanying drawings of the embodiments.

[0028] Figure 1 This diagram illustrates a flowchart of a block-diffusion approximate linear attention calculation method for a gated incremental network basic model according to an embodiment of the present invention. Figure 2 This diagram illustrates a process for calculating block-diffusion approximate linear attention for a gated incremental network basic model according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, a block-diffusion approximate linear attention calculation method for a gated incremental network basic model includes: First, in step 101, intermediate parameters are generated using the original projection structure. The GDN layer to be modified and its input hidden state H in the basic model are obtained. Based on the original projection structure of the GDN layer, query Q, key K, value V, gating coefficient β, attenuation parameter g, and necessary gating normalization parameters are generated. L2 normalization is performed on Q and K, and Q can be scaled according to the square root of the query vector dimension. Next, in step 102, approximate expansion is performed. The causal recursive state update of GDN is approximated as explicit pairwise aggregation within a finite sliding window. Specifically, this involves obtaining the block diffusion boundary information of the current training segment, including Noised, Clean segmentation location, sample segment boundary, block size B, sliding window size W, and the block number corresponding to each token. The original recursive state update of GDN is approximated as explicit pairwise aggregation based on Q, K, V, β, and a decay term within a finite sliding window. The purpose of approximate expansion is not to turn the base model into a regular sliding window attention model, but to obtain an approximate kernel with acceptable explicit visibility control while preserving the GDN projection, gating, decay, and interface form. Next, in step 103, a block diffusion visibility control is embedded. The block diffusion visibility control is embedded in the explicit pairwise aggregation. Construct an approximate kernel with a unified computational form; Next, in step 104, path-specific aggregation is performed. For Clean paths, only causal sliding windows that do not cross sample boundaries are accessed, i.e., only the Clean Token at the current position and before it is accessed. For Noised paths, Clean-Prefix aggregation and Same-Block bidirectional aggregation are performed: the former only accesses the Clean Prefix before the current noise block, and the latter only accesses the positions within the Current Noised Block that satisfy the bidirectional local window condition; at the same time, access to the Current Clean Block, Future Clean Block, other Noised Blocks, and other samples is prohibited. The Clean and Clean-Prefix paths use directed cumulative decay difference, while the Same-Block path uses the negative of the absolute value of the cumulative decay difference. If the original GDN layer contains a causal depthwise convolution or a similar local feature generation module, then the convolution history that crosses the starting point of the Noised Block in the Noised branch is repaired using Clean-Prefix.

[0029] Finally, in step 105, the outputs of each path are merged. The Clean Causal-Sliding-Window output, the Noised Clean-Prefix output, and the Noised Same-Block output are calculated separately. The two sub-path outputs of the Noised output are added together, and then merged with the Clean path output in the original training segment order. The merged output is then fed into subsequent network layers or loss functions through gating normalization and output projection compatible with the original GDN layer.

[0030] This enables a hierarchical transformation of GDN approximate kernel and block diffusion visibility, which is not a repetitive description of ordinary block diffusion training objectives, ordinary global attention masks, or ordinary sliding window attention.

[0031] In native GDN, token mixing is typically implemented using a recursive state over time. In one embodiment of this invention, the recursive state at position t is represented as: , The output is represented as: , in, , , These represent the query, key, and value vectors obtained by projection from the GDN layer, respectively. Indicates the gating or write coefficient. This represents the state transition terms associated with decay and Delta updates. In a common form, It can be written as: , Expanding the above recursive relationship, we can see that the output at position t depends on the cumulative state transition of historical position i: , It can be seen that this form has a strict left-to-right recursive dependency, and historical information is compressed into a single state. It is suitable for ordinary autoregressive modeling, but when it is necessary to introduce bidirectional visibility within the block later, the state... It is difficult to explicitly distinguish whether a certain part of the history comes from the Clean Prefix or the Noised Block, and writing bidirectional information from the current Noised Block into the same recursive state would disrupt the precise control over the source of visibility during block diffusion training. Therefore, in one embodiment of this invention, the recursive state of the GDN is explicitly approximated and expanded within a finite sliding window. Instead of directly maintaining a single recursive state that carries all the history, the pairwise interaction of the Query Token and Key Token is explicitly calculated within the window, while retaining the original projection, gating, and decay parameters of the GDN, ensuring that this pairwise aggregation remains compatible with the parameter structure of the original GDN layer.

[0032] Specifically, the state transition product term is approximated as an exponential decay kernel determined by the cumulative decay: , in, It is the identity matrix. The cumulative decay amount is obtained by accumulating the decay parameters per token: , In one embodiment of the present invention, It is obtained from the attenuation-related parameters, time-step modulation parameters, or gated projection results in the original GDN layer. For example, when the original model contains logarithmic attenuation parameters and time-step bias, the attenuation can be generated in the following form: , It should be understood that in other embodiments of the present invention, other attenuation parameterization methods compatible with the original GDN layer may also be used in different GDN variants.

[0033] Given a finite window size of W, a causal sliding window is defined for the t-th token in both the Clean causal path and the Noised Clean-Prefix path; a bidirectional local window, constrained by both the same-block condition and positional distance, is defined for the Noised Same-Block path. , The GDN output can then be approximated as pairwise aggregation within an explicit sliding window: , in, For gating or writing coefficients, The cumulative decay is obtained by accumulating the decay parameters for each marker. For the first Attenuation parameters at each location, This represents the relative attenuation term. The label obtained by projecting the gated incremental network layer The query vector, The first layer obtained by projecting the gated incremental network layer A key vector at each position, The first layer obtained by projecting the gated incremental network layer A vector of values ​​at each position.

[0034] To avoid numerical instability caused by an excessively large or small exponent term, in one embodiment of the present invention, the relative decay term is further truncated. Let the truncation threshold be c, then the relative decay term can be expressed as: Based on this, the corresponding approximate kernel is: , This approximate expansion transforms the previously implicit GDN recursive state into an explicit Token-Pair Kernel. It still uses the original Q, K, V, β, and decay parameters of the GDN layer, thus facilitating the inheritance of existing weights and inductive biases from the base model. Furthermore, each Query-Key pair can be explicitly selected or masked, providing a computational platform for subsequent embedding block diffusion visibility control.

[0035] When the window size W is a fixed parameter that does not grow with the sequence length N, the time complexity of explicit pairwise aggregation is O(NW), and thus grows linearly with respect to N; if W grows with N, it is not recommended that it maintain linear complexity with respect to N.

[0036] In practical applications, for Packed Training scenarios, it's necessary to avoid cumulative decay propagating across samples. Let the boundary of the Packed Sequence be: , Therefore, for a local position t within the m-th sample, the cumulative attenuation should be recalculated within that sample: , As a necessary boundary constraint when using the GDN approximation kernel for large-scale packed training, it can ensure that neither sliding window aggregation nor cumulative decay propagates across sample boundaries.

[0037] In the GDN explicit approximation kernel obtained by the method described above, the calculation of each output token can be represented as a weighted aggregation of several Query-Key pairs within the window. Embedding the visibility rules required for SDAR2 or block diffusion training into this approximation kernel allows it to support Clean Block path, Noised Clean-Prefix path, and Noised Same-Block path while retaining the form of the GDN approximation kernel.

[0038] Let block(t) be the block number to which token t belongs. When bB≤t<(b+1)B, let block(t)=b, where B is the block size. For the b-th block, let Clean Block be... Noised Block is , Previous Clean Prefix is: , In one embodiment of the present invention, the Clean and Noised segments can be ordered as follows: It packages the data in a formal way and establishes a correspondence between Noised and Clean positions through Segment_Bounds. This serves the implementation of the approximate kernel and embeds block diffusion visibility in the GDN approximate kernel.

[0039] An example of unrestricted packing is [x_t, x_0]: x_t is located in [start, mid), x_0 is located in [mid, end), and the two have the same length; the logical position is mapped as logical_pos(x_t[j])=logical_pos(x_0[j])=j, and the window, block number and cumulative decay are reset at each (start, mid, end) boundary.

[0040] For Clean Query, perform causal sliding window aggregation, as shown in Table 1, so that it only accesses Clean Tokens at the current position and before that in the same sample that fall into the window, and does not access any future Clean Tokens, Noised Tokens or other samples. Its visible set can be represented as: , For Noised Query, as shown in Table 1, it is allowed to access the Previous Clean Prefix and Current Noised Block, but not allowed to access Current Clean Block, Future Clean Block and other Noised Blocks, so as to avoid training target leakage. The visible set of it can be expressed as: .

[0041] Table 1 Window constraints are defined respectively for different paths: the Clean causal path and the Noised Clean-Prefix path satisfy t-W+1≤i≤t; the Noised Same-Block path satisfies block(i)=block(t) and |i-t|<W. When W is not less than the block size B, the Same-Block path can cover all positions of the current noised block. The final valid Noised set is the union of the Clean-Prefix causal window and the Same-Block bidirectional local window.

[0042] .

[0043] embed into the aforementioned GDN approximate kernel, a unified calculation form can be obtained: , , It can be seen that the aforementioned method does not simply apply a mask on ordinary Transformer Attention, but writes the block diffusion visibility into the whole process of QK, attenuation, β gating and V aggregation of the GDN approximate kernel.

[0044] Based on this, for the Clean path, it maintains the Next-Token causal semantics of the original gated incremental network layer, and only aggregates the current position and the preceding Clean Tokens within a limited sliding window, and does not perform bidirectional aggregation within the current Clean Block.

[0045] In an embodiment of the present invention, the Noised path is further decomposed into two subpaths. The first subpath is the Clean-Prefix path, which is used to read the Previous Clean Prefix as cross-block history: , The second sub-path is the Same-Block path, used for bidirectional local denoising between tokens within the current Noised Block: , The final Noised output is: .

[0046] By using the twin-path structure described above, we can avoid mixing the Clean Prefix history and CurrentNoised Block local information into a single recursive state, while also avoiding the need to completely replace the GDN base model with global Transformer Attention.

[0047] To align the attenuation term with the aforementioned visibility sources, in one embodiment of the invention, a path-specific cumulative attenuation is further constructed. For position r in the Clean path, the attenuation can be accumulated in causal order: , For position r within the Noised Block, the boundary accumulation of the Clean Prefix is ​​used as the cross-block offset, and the local accumulation of the Noised Block is used within the current Noised Block: , Therefore, for the Noised Clean-Prefix path, the cumulative amount on the query side uses a combination of Clean Prefix boundary accumulation and Noised local accumulation, while the cumulative amount on the key side uses Clean Prefix accumulation. This path employs a directed cumulative attenuation difference. For the Same-Block path, both the query and key use the local cumulative amount within the Noised Block, and the negative of the absolute value of their difference is used to form a bidirectional symmetrical attenuation. This avoids incorrect writing of Noised history into the cross-block attenuation state.

[0048] In one embodiment of the present invention, if the original GDN layer contains a causal depthwise convolution or a similar local feature generation module, modifying the attention visibility alone is insufficient to guarantee semantic consistency. Ordinary causal convolution causes the noisy, labeled cross-block convolution history to originate from previous noised history, while block-spread semantics requires the cross-block history to originate from the Clean Prefix. Therefore, historical patching is performed on the noised branches during the local convolution stage.

[0049] Let the kernel size be K and the convolution weights be... For position r within the Noised Block, construct the patched historical input. Specifically, when rl is still within the current Noised Block, Take the Noised input, and when rl crosses the starting point of the current Noised Block, Taking the corresponding Clean Prefix as input, the block diffusion-aware convolution can be written as: , in, This indicates the activation function used in the original layer. This patch ensures that local convolutional features, mixed cumulative decay, and attention visibility remain consistent within the same block diffusion semantics.

[0050] This completes the embedding of block diffusion visibility into the GDN explicit approximation kernel. Its core is not a typical Clean / Noised two-branch process, but rather a unified constraint on historical origins within the paired aggregation, cumulative decay, and local convolutional features of the GDN approximation kernel.

[0051] Expanding the GDN recursive state into explicit window pair aggregations and further embedding block diffusion visibility transforms the computation graph from a single recursive path into a Clean Causal-Sliding-Window path, a Noised Clean-Prefix path, a Noised Same-Block path, and an optional local convolutional patching path. Directly implementing this using segment-by-segment Python loops and ordinary tensor operations easily generates numerous small operators, tensor concatenations, data copies, and CPU / GPU synchronization points, leading to decreased training throughput and GPU power fluctuations. Therefore, in one embodiment of this invention, an optimization is provided for a training method based on the aforementioned GDN approximate block diffusion kernel.

[0052] First, it is preferable to pre-compute segment boundary information on the data side or during the batch construction stage. For training batches using PackedSequence, the corresponding noise and clean segments can be stored in the physical order of [x_t, x_0], and segment_bounds=(start, mid, end) can be used to represent the sample start point, noise and clean segmentation positions, and sample end point; logical position mapping is used to make the positions in x_t and the corresponding positions in x_0 share the same block number. The cumulative decay is recalculated at each sample boundary and cross-sample reading is prohibited. Each network layer directly reads the above metadata during forward computation to reduce CPU and GPU synchronization points.

[0053] Secondly, it is preferable to align and group training samples by block size and effective sequence length. For Noised Same-Block paths and local convolutional patching paths, if the sample length is not an integer multiple of the block size, additional padding, boundary checks, or index patching are required. By aligning sample lengths to the block size as much as possible and grouping fragments with the same or similar shapes for execution, general indexing logic can be reduced, invalid padding can be reduced, and the batch workload of GPU kernel functions can be improved.

[0054] Third, it is preferable to use custom GPU kernel functions to implement the core path. For the Clean Causal-Sliding-Window path, QK, decay, beta gating, visibility determination, and V aggregation within the causal window can be completed within a single kernel function. For the Noised Clean-Prefix path, the query can come from the Noised Block, and the key and value can come from the Clean Prefix preceding the current block. For the Noised Same-Block path, bidirectional QK, exponential decay, beta gating, and V aggregation are performed within the current Noised Block. In one embodiment of this invention, the above kernel functions can be implemented on a CUDA graphics processor using Triton. Other accelerated backends should be used when the same visibility set, decay, and gradient calculation can be achieved.

[0055] Fourth, it is preferable to kernelize both forward and backward propagation. If only a custom kernel function is used for forward propagation and backward propagation reverts to ordinary tensor operations, training will still result in a large number of intermediate tensor writes, atomic writes, or small operator scheduling. To address this, the gradient calculations corresponding to Q, K, V, β, cumulative decay, and visibility mask can be merged into the backward GPU kernel function. In terms of implementation, different tile sizes can be selected based on the Head dimension, Value dimension, Block_Size, and Window_Size to balance memory usage and computational throughput.

[0056] Fifth, it is preferable to fuse output merging and local convolutional patching using nearest-neighbor scheduling. Since the Noised output is obtained by adding the Clean-Prefix sub-path and the Same-Block sub-path, writing each sub-path back to global memory separately and then summing them individually would result in additional read / write operations and synchronization. Where feasible, the sum of the two sub-paths can be placed in the same output buffer, or they can be grouped and scheduled continuously according to the same segment, thereby reducing global memory round trips. For local convolutional patching, the cross-block history replacement logic can also be merged with the convolutional output correction, avoiding the need to generate a complete ordinary convolution result first and then patch it token by token.

[0057] Sixth, the above optimizations are preferably limited to engineering implementations serving the GDN approximate block diffusion kernel. That is, when the GDN recursive state is explicitly expanded and block diffusion visibility is further added, in order to reduce computational fragmentation caused by Clean, Noised path splitting, Same-Block bidirectional aggregation and mixed cumulative decay, a method of segmented pre-computation, block alignment, group execution and custom GPU kernel function fusion is adopted.

[0058] Through the above optimizations, while maintaining GDN base model interface compatibility and precise control of block diffusion visibility, the training scheduling fragmentation caused by explicit approximate kernels can be reduced, thereby improving throughput stability and GPU utilization when large models continue to be trained.

[0059] In a non-restrictive implementation, the diffusion block size B is 4, the sliding window size W is 512, and the input uses a PackedTHD layout with [x_t, x_0] pairs packed together. The core forward and reverse paths are implemented using Triton / CUDA kernel functions. These parameters are for illustrative purposes only and do not constitute a limitation on the scope of protection.

[0060] Based on the block diffusion approximate linear attention calculation method described above. Figure 3 This diagram illustrates the structure of a model training device according to an embodiment of the present invention, as shown below. Figure 3 As shown, a model training device includes a training sample construction unit 300, a gated incremental network layer projection unit 301, a state unrolling unit 302, a visibility control unit 303, a path calculation unit 304, an output merging unit 305, a loss calculation unit 306, and a parameter update unit 307. The training sample construction unit 300 generates training samples containing noisy segments and corresponding clean segments, and generates sample boundaries, noise and clean segmentation locations, and block numbers. The gated incremental network layer projection unit 301 is used to obtain the Q, K, V, β gating parameters and attenuation parameters corresponding to the gated incremental network layer. The state unrolling unit 302 is used to project the gated incremental network layer... The causal recursive state update of the layer is approximately expanded into explicit pairwise aggregation within a finite sliding window. The visibility control unit 303 is used to construct a clean causal sliding window, a noisy clean prefix window, and a noisy bidirectional local window within the same block, and embeds the corresponding visibility control terms into the explicit pairwise aggregation. The path calculation unit 304 is used to perform causal sliding window aggregation on clean labels, perform clean prefix aggregation and bidirectional local aggregation within the same block on noisy labels, and use path-specific cumulative attenuation. The output merging unit 305 is used to merge the outputs of each path. The loss calculation unit 306 calculates the loss based on the block diffusion training objective, and the parameter update unit 307 updates the model parameters based on the loss.

[0061] Compared to the native GDN recursive gated linear attention, the method of this invention rewrites the implicit recursive state into explicit sliding window pairwise aggregation, enabling visibility to be... Precise control enables bidirectional modeling within the Noised Block and Clean Prefix cross-block history.

[0062] Compared to directly replacing with global Transformer Attention or ordinary sliding window Attention, this invention preserves as much of the projection, gating, decay, and interface form of the GDN base model as possible. Compared to the only block diffusion or SDAR training paradigms, this invention provides block diffusion visibility for GDN layers, path-specific cumulative decay, Same-Block bidirectional approximate linear aggregation, and local convolutional patching. Compared to the ordinary GDN→SW approximate kernel, this invention further constrains the historical source under block diffusion semantics, enabling Clean Causal-Sliding-Window paths, Noised Clean-Prefix paths, Noised Same-Block paths, and local convolutional features to operate under the same visibility definition.

[0063] It should be understood that, in the embodiments of the present invention, the GDN base model can be a model in which all layers use GDN, or a hybrid model in which some layers use GDN and some layers use global attention or other token mixing layers. In the embodiments of the present invention, the sliding window size and Block_Size can be fixed values, or they can be dynamically adjusted by layer, by sample length, or by training stage. The generation method of Noised Tokens can be occlusion, random replacement, discrete noise, multi-stage noise, or other perturbation methods compatible with block diffusion training. The decay function, β-gating form, and normalization method in the GDN approximation kernel can be equivalently adjusted according to the specific GDN variant of the base model. The Same-Block path can be bidirectional within the entire block, or it can be bidirectional with local windows, sparse bidirectional, or grouped bidirectional, as long as it serves the local denoising modeling within the Noised Block. Local convolutional patching can be extended to other local feature modules, such as local convolution, state space local modules, or gated feedforward local modules. The GPU kernel function can be implemented using Triton, CUDA, HIP, XLA, or other accelerated backends.

[0064] The methods described above can be used for continued training, supervised fine-tuning, denoised pre-training, instruction data distillation, agent trajectory distillation, and long context adaptation training.

[0065] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A block-diffusion approximate linear attention calculation method for a gated incremental network basic model, characterized in that, comprising: obtaining a query, a key, a value, a gating coefficient and a decay parameter corresponding to a gated incremental network layer; approximately expanding the causal recursive state update of the gated incremental network layer into explicit pairwise aggregation within a finite sliding window; embedding a block diffusion visibility control term into the explicit pairwise aggregation to construct an approximate kernel in a unified calculation form; based on the block diffusion visibility control term, performing causal sliding window aggregation on clean tokens, and performing clean prefix aggregation and bidirectional local aggregation within a current noise block on noise tokens respectively; merging outputs of each path, wherein the output result maintains interface compatibility with the gated incremental network layer.

2. The block diffusion approximate linear attention calculation method as described in claim 1, characterized in that, approximately expanding the causal recursive state update of the gated incremental network layer into explicit pairwise aggregation within a finite sliding window comprises: Multiply the query vector by After scaling, the vector is then L2 normalized with the key vector, where The dimension of the query vector; explicitly calculating the pairwise interaction between query tokens and key tokens within the window, while retaining the original projection, gating parameters and decay parameters of the gated incremental network.

3. The block diffusion approximate linear attention calculation method as described in claim 2, characterized in that, said explicitly calculating the pairwise interaction between query tokens and key tokens within the window comprises: approximating a state transition product term as an exponential decay kernel determined by a cumulative decay amount: when the finite window size is W, defining a causal sliding window set of a t-th token for a clean causal path and a noise clean prefix sub-path, and defining a bidirectional local window constrained by both block boundaries and position distances for a noise same-block sub-path: ， then the output of the gated incremental network is approximated as pairwise aggregation within an explicit sliding window: ， in, For gating or writing coefficients, The cumulative decay is obtained by accumulating the decay parameters for each marker. For the first Attenuation parameters at each location, This represents the relative attenuation term. The label obtained by projecting the gated incremental network layer The query vector, The first layer obtained by projecting the gated incremental network layer A key vector at each position, The first layer obtained by projecting the gated incremental network layer A vector of values ​​at each position.

4. The block diffusion approximate linear attention calculation method as described in claim 3, characterized in that, said explicitly calculating the pairwise interaction between query tokens and key tokens within the window further comprises: truncating the relative decay term and expressing it as: ， in, This is the truncation threshold; the path-specific decay kernel is expressed as: ， in Used for clean causal paths and noisy clean prefix subpaths. Used for noise-prone dual-line paths.

5. The block diffusion approximate linear attention calculation method as described in claim 1, characterized in that, said block diffusion visibility control term is: in, For clean blocks, For noise blocks, For the visible set of clean queries, The visible set for noise queries.

6. The block diffusion approximate linear attention calculation method as described in claim 1, characterized in that, said respectively performing clean prefix aggregation and bidirectional local aggregation within the current noise block on noise tokens comprises: constructing a clean prefix sub-path, and aggregating the clean prefix before the current noise block as cross-block history; constructing a same-block sub-path to perform bidirectional local denoising aggregation between tokens within the current noise block, wherein visible positions of the same-block sub-path satisfy block(i)=block(t) and |i-t|<W, and when W≥B, the same-block sub-path covers all positions of the current noise block, wherein B is a noise block size; adding the output of the clean prefix sub-path and the output of the same-block sub-path to obtain a final output of a noise token.

7. The block diffusion approximate linear attention calculation method as described in claim 6, characterized in that, the clean causal path, the clean prefix sub-path and the same-block sub-path adopt a path-specific cumulative decay mechanism: the clean causal path and the clean prefix sub-path adopt a directional difference between a query position cumulative decay amount and a key position cumulative decay amount; the same-block sub-path adopts an opposite number of an absolute value of a difference between the query position cumulative decay amount and the key position cumulative decay amount, so as to form symmetric exponential decay suitable for bidirectional positions.

8. The block diffusion approximate linear attention calculation method as described in claim 6, characterized in that, the same-block sub-path performs QK similarity calculation, exponential decay calculation, β gating calculation and V aggregation calculation within the current noise block.

9. The block diffusion approximate linear attention calculation method as described in claim 1, characterized in that, when the gated incremental network layer comprises a causal convolution or a local feature module, reading a noise input from historical positions inside the current noise block, and reading a corresponding clean prefix input from historical positions crossing a start point of the current noise block.

10. A model training device, characterized in that, comprising: The training sample construction unit is configured to construct training samples containing noisy segments and corresponding clean segments, and generate sample boundaries, noise and clean segmentation locations, and block numbers. The gated incremental network layer projection and state unrolling unit is configured to obtain the query, key, value, gating coefficient and decay parameter, and approximate the causal recursive state update as an explicit pairwise aggregation within a finite sliding window; The visibility control unit is configured to construct a clean causal sliding window, a noisy clean prefix window, and a noisy co-block bidirectional local window, and embed the corresponding visibility control terms into the explicit pairwise aggregation to construct an approximate kernel with a unified computational form; The path calculation unit is configured to perform causal sliding window aggregation on clean tags, clean prefix aggregation and co-block bidirectional local aggregation on noisy tags, and to employ path-specific cumulative attenuation. The output merging, loss calculation, and parameter update unit is configured to merge the outputs of each path, obtain the model output through an output interface compatible with the gated incremental network layer, calculate the loss based on the block diffusion training objective, and update the model parameters based on the loss.

Images