Transform optimization method and system based on token graph query propagation mechanism

Abstract

This invention relates to the field of deep learning technology, particularly to a Transformer optimization method and system based on a token graph-based query propagation mechanism. The method comprises an input processing layer, a dynamic graph construction layer, a propagation control layer, a computational optimization layer, a feedback adjustment layer, and an output integration layer. This invention captures the contextual similarity or relationship between tokens by constructing a token-graph (dynamic or static structure) and introduces a stochastic propagation of Q vectors mechanism. Only the seed token generates a complete Q vector, while the remaining tokens generate approximate Q vectors through graph propagation. Furthermore, K and V are projected only onto the central node, and a local error control feedback module is included. This solves the problems of computational resource constraints, memory limitations, and accuracy stability in long sequence processing of Transformers. It provides a practical and feasible technical path for its application in complex tasks such as long document understanding, multi-turn dialogue, and code generation.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a Transformer optimization method and system based on a Token graph-based Query propagation mechanism. Background Technology

[0002] The Transformer architecture has become the dominant model in natural language processing, computer vision, and multimodal tasks, achieving significant breakthroughs in areas such as machine translation, text generation, and image recognition. However, as application scenarios increasingly demand longer sequence lengths, the Transformer has revealed serious limitations when processing long sequences. These shortcomings mainly stem from the multi-head attention computation of its core self-attention mechanism, which requires independently generating Query (Q, query vector), Key (K, key phasor), and Value (V, value vector) vectors for each token, resulting in a quadratic increase in computational complexity of O(n log n). 2 d), where n is the sequence length and d is the model dimension. This leads to problems such as computational resource explosion, excessive memory consumption, and inference latency when dealing with long documents (such as legal provisions, academic papers), multi-turn dialogue history, or code files in actual deployments.

[0003] To alleviate the above problems, existing technologies have proposed various optimization schemes, but they still have the following shortcomings:

[0004] The bottleneck of quadratic complexity: The traditional Transformer's QKV projection and attention matrix calculation require a complete matrix multiplication operation for each token. As the sequence length n increases, the computational cost and memory requirements quadruple. Existing optimization schemes such as Longformer or BigBird use fixed-window sparse attention, which alleviates some of the problems, but they still rely on the window size. Information outside the window is ignored, and they cannot truly capture long-range dependencies.

[0005] Information redundancy and static processing limitations: Existing clustering methods (such as ClusTR or ClusterKV) reduce redundancy by simply grouping and sharing QKV computation results. However, the clustering process is often static and lacks dynamic relationship modeling between tokens, leading to decreased accuracy with variable inputs (such as varied dialogues or noisy data). Furthermore, these methods do not consider the contextual similarity or relationships of tokens, resulting in insufficient information propagation and preventing true adaptive optimization.

[0006] Insufficient error control and hardware compatibility: Existing solutions such as FACT's eager discarding support hardware pipeline discarding irrelevant computations, but lack intelligent error feedback mechanisms, which can easily introduce precision loss. Furthermore, these methods do not fully utilize the structured relationships between tokens (such as semantic or positional edges), resulting in inefficiency during hardware deployment and an inability to dynamically adjust to different loads.

[0007] Therefore, we propose a Transformer optimization method and system based on a Token graph-based Query propagation mechanism to solve the above problems. Summary of the Invention

[0008] This invention addresses the shortcomings of existing Transformer architectures, such as secondary complexity bottlenecks, information redundancy, static processing limitations, insufficient error control, and inadequate hardware compatibility. It provides a Transformer optimization method and system based on a Token graph-based Query propagation mechanism.

[0009] This invention is achieved through the following technical solution:

[0010] A Transformer optimization system based on a Token Graph-based Query Propagation Mechanism, comprising:

[0011] Input processing layer: used to encode the input sequence to generate the initial hidden state;

[0012] Dynamic graph construction layer: used to construct the token graph and sparse adjacency matrix based on the initial hidden state mentioned above;

[0013] Propagation control layer: used to select seed nodes, generate seed queries, and propagate the full query matrix through the token graph;

[0014] Computational optimization layer: used for sparse projection seeds (Key and Value), and computes attention output;

[0015] Feedback adjustment layer: used to detect propagation errors, selectively recalculate, and adaptively adjust parameters;

[0016] Output integration layer: used for residual connections, layer normalization, and feedforward network processing to generate the final output.

[0017] In a preferred embodiment of the present invention, the input data in the input processing layer originates from the output of the word segmenter.

[0018] In a preferred embodiment of the present invention, the dynamic graph construction layer includes a similarity calculation module, a matrix sparsification module, and a graph generation module, wherein the sparsification in the matrix sparsification module is based on the number of neighbors retained for each node, which is related to the logarithm of the sequence length.

[0019] In a preferred embodiment of the present invention, the propagation control layer includes a seed node selection module, a seed query generation module, and a query propagation module. The propagation control layer receives the output of the dynamic graph construction layer and realizes the transformation from full computation to sparse propagation by selecting seed nodes and propagating queries. The propagation uses message passing with a fixed number of steps.

[0020] In a preferred embodiment of the present invention, the computation optimization layer includes a sparse KV projection module and an attention computation module, wherein the computational complexity of the attention is O(n×S×d), where S is the number of seeds.

[0021] In a preferred embodiment of the present invention, the feedback adjustment layer includes an error detection module, a selective recalculation module, and a parameter adaptive module, wherein the error is calculated using the L2 norm.

[0022] In a preferred embodiment of the present invention, the output integration layer includes a residual connection and layer normalization module and a feedforward network module, and the output integration layer is compatible with the traditional Transformer.

[0023] In a preferred embodiment of the present invention, a parameter initialization module is further included, which is used to initialize the projection matrix, seed ratio, sparsity coefficient, propagation steps and error threshold.

[0024] A method for optimizing a Transformer system based on a token graph-based query propagation mechanism includes the following steps:

[0025] S1. Input Encoding Stage: Token encoding and position encoding are performed on the input sequence to generate the initial hidden state;

[0026] S2. Graph Construction Stage: Based on the initial hidden state, calculate the cosine similarity or scaled dot product as the similarity value to form a similarity matrix between tokens. Then, according to the number of the most important neighbors retained by each node, perform matrix sparsification to generate a sparse adjacency matrix. At the same time, define the node set, edge set, and weight function based on the sparse adjacency matrix to construct the token graph.

[0027] S3, Seed Selection Stage: Select a set of seed nodes from the Token graph. The selection strategy includes uniform random sampling, degree centrality-based selection, or node position-based selection. And generate a complete query vector by linearly projecting only the initial hidden state of the seed nodes.

[0028] S4, Query Propagation Phase: The Query vector of the seed node is iteratively propagated through the sparse adjacency matrix using the propagation function to generate the Query matrix of the entire sequence. The number of propagation steps is a preset value to ensure full graph coverage.

[0029] S5, KV projection stage: Key and Value matrices are generated only for the initial hidden state projection of the seed nodes; projection is omitted for non-seed nodes.

[0030] S6. Attention Calculation Stage: Calculate the dot product of the Query matrix and the seed Key matrix, apply softmax normalization, and then sum the product with the seed Value matrix using weighted summation. Perform attention calculation and generate attention output.

[0031] S7. Feedback Control Phase: Calculate the norm error between the propagated Query and the reference Query; recalculate the accurate Query for nodes where the error exceeds the threshold; adaptively update the threshold based on the error gradient.

[0032] The beneficial effects of this invention are:

[0033] 1. This invention utilizes a graph structure to achieve dynamic interactive propagation of queries, overcoming the limitations of static grouping in existing clustering methods and reducing computational complexity from O(n^2) to O(n^2). 2 d) Reduced to near-linear O(n × k + S × d) 2 ), where S is the number of memory slots or seeds of a fixed size, n is the sequence length, k is the average number of neighbor nodes (average degree of the graph), and d is the model's hidden dimension. This solves the double-growth bottleneck and also supports processing arbitrarily long sequences;

[0034] 2. This invention introduces a small feedback module to detect tokens with high propagation errors in real time and accurately calculate their Q / K / V, ensuring that the model accuracy is comparable to that of the traditional Transformer. At the same time, this mechanism simulates human attention allocation, improving robustness under noise or complex inputs and solving the problems of "instant forgetting" or accuracy loss.

[0035] 3. This invention supports the generation of complete QKV with eager discarding of irrelevant tokens, seamlessly integrates with existing hardware pipelines, indirectly reduces the number of KV dot-products, is suitable for edge devices and large-scale deployments, solves the problem of insufficient system compatibility in existing technologies, and provides more efficient resource allocation;

[0036] In summary, by constructing a token-graph (dynamic or static structure) to capture contextual similarity or relationships between tokens, and introducing stochastic propagation of Q vectors, a complete Q is generated only for the seed token, while the remaining tokens generate approximate Q vectors through graph propagation. Furthermore, K and V are projected only onto the central node, and a local error control feedback module is included. This approach solves the computational resource constraints, memory limitations, and accuracy stability issues of Transformer in long sequence processing, providing a practical technical path for its application in complex tasks such as long document understanding, multi-turn dialogue, and code generation. At the same time, this approach maintains the core advantages of Transformer, such as parallel computing and attention mechanisms, and ensures full compatibility with existing ecosystems (such as BERT and GPT series models) through an end-to-end trainable design. Attached Figure Description

[0037] Figure 1 This is a block diagram of the Transformer optimization system based on the Token Graph Query Propagation Mechanism of the present invention. Detailed Implementation

[0038] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0039] like Figure 1 The Transformer optimization system shown is based on a token graph-based query propagation mechanism. It employs a six-layer architecture, achieving a significant reduction in computational complexity through a hierarchical processing flow. The system as a whole follows the core design principle of "build-propagate-feedback," with each layer undertaking specific functional responsibilities, including:

[0040] The input processing layer is responsible for processing the original input sequence X=[x1, x2, ..., x...]. n After token encoding and position encoding, the initial hidden state H is generated. 0 ∈R n×d ;

[0041] Wherein, the original input sequence X=[x1, x2, ..., x n The data type of ] is an integer array, and the data comes from the output of the tokenizer. n is the sequence length, and x is the data source. i This represents the index of the i-th token in the sequence;

[0042] Token encoding: E∈R v×d Where v represents the vocabulary size, d represents the embedding dimension / hidden dimension, and R is the real number field. E[X] represents the embedding matrix of all tokens;

[0043] Location encoding: PE∈R n×d Where n represents the sequence length and d represents the hidden dimension, in the batch processing scenario, the dimension of the position encoding is expanded to B×n×d, where B represents the batch size;

[0044] The initial hidden state H output by the input processing layer 0 ∈R n×d H is obtained by adding the token embedding matrix E[X] and the position encoding matrix PE element by element, i.e. 0 =E[X]+PE, where n×d is the dimension, and the real matrix space;

[0045] All the above data can be obtained by calculation using existing technologies or algorithms. Token encoding and location encoding can both be implemented using existing mature technologies. The data type is float32, and after normalization, the value range is usually between [-3, 3], which is used as the input for subsequent modules.

[0046] The following table analyzes memory usage:

[0047]

[0048] The following is a calculation example in this layer:

[0049] Given the calculation conditions (v=30000, n=512, d=768, B=32), the memory usage of the embedding matrix E is calculated as v×d×4=30000×768×4=92160000 bytes, and using 1MB=1000000 bytes, the memory usage is calculated to be 92.16MB.

[0050] Output H 0 The memory usage is calculated as n×d×4=512×768×4=1572864 bytes, which is approximately 1.57MB of memory;

[0051] The memory usage of the batch process is calculated as B×n×d×4=32×512×768×4=50331648 bytes, which is approximately 50.33MB of memory;

[0052] Since E is a shared parameter, the total memory calculation does not repeat the calculation of a single sample, resulting in 92.16MB + 50.33MB, which is approximately 142.49MB.

[0053] The dynamic graph construction layer is responsible for constructing the input hidden state H.0 It is converted into a token graph structure G=(V,E,W) and a sparse adjacency matrix A, including a similarity calculation module, a matrix sparsification module and a graph generation module;

[0054] The similarity calculation module is based on the initial hidden state H output by the input processing layer. 0 Calculate the similarity value Sim(i,j) between token i and token j. The calculation method of Sim(i,j) is selected from either cosine similarity [-1, 1] or scaled dot product [-∞, +∞]. After actual calculation, it is approximately [-100, 100], which becomes [0, 1] after passing through sigmoid, forming an n×n similarity matrix Sim.

[0055] The matrix sparsification module generates a sparse adjacency matrix A based on the similarity matrix Sim, expressed as A = Sparse(Sim, k). The specific operation is as follows: First, determine the number of neighbors k that each token node needs to retain. k is determined by the formula... The calculations show that k_max is the maximum number of neighbors, with data from the model hyperparameter configuration; typical values ​​can be 64 or 128. α is the sparsity coefficient, also from the hyperparameter configuration, which can be tuned using the validation set; typical values ​​can be 5-20, used to control the relationship between sparsity and sequence length. log(n) is the natural logarithm of the sequence length, where n is the current sequence length (from the input processing layer). Using logarithms ensures the sparsity of long sequences and avoids k growing linearly with n. The first step is to round down. The second step involves sorting each row of the similarity matrix Sim by value, selecting the k largest Sim(i,j) values, retaining their original values, and setting the remaining Sim(i,j) values ​​to 0. This completes the transformation from the similarity matrix Sim to the sparse adjacency matrix A, reducing the connection number of the sparse adjacency matrix A from O(n) of the similarity matrix Sim. 2 The complexity is reduced to O(n·k), which ensures that each token can establish connections with at most k other tokens;

[0056] The graph generation module constructs a token graph structure G=(V,E,W) based on the sparse adjacency matrix A, explicitly defining the node set V, the edge set E, and the edge weight function W. The data structure of the token graph structure G is a graph object, which is used to pass to the propagation control layer to realize query propagation.

[0057] Where the node set V = {v1, v2, ..., v} n}, each node v iFor each token, the number of nodes in the node set V is |V|=n (the number of nodes is equal to the sequence length). The data is directly determined from the token sequence, with one node at each position.

[0058] edge set The set E consists of the positions of all non-zero elements in the sparse adjacency matrix A. The data is obtained by traversing the sparse matrix A and collecting all non-zero positions. The number of elements in the edge set E is |E|≤n×k (each node has at most k outgoing edges).

[0059] The edge weight function W:E→R is used to map each edge in the token graph structure G to a real weight, satisfying W(i,j) = A ij ,in ;

[0060] The following is a calculation example in this layer:

[0061] Set the calculation conditions (n=512, k=32, d=768);

[0062] Similarity matrix Sim: Full storage 512 2 ×4 = 1048576 bytes = 1MB, block calculation only requires one line at a time, 512 × 4 = 2KB;

[0063] According to the formula Let α = 10, k_max = 64. For the simplified example, k=32;

[0064] Sparse adjacency matrix A: According to the sparsification formula, each row of A has at most k non-zero values, and the total number of non-zero elements is at most n×k=512×32=16384. Sparse storage is 16384×4=65536 bytes≈64KB, dense storage is 512 bytes≈64KB. 2 ×4=1MB, memory saving: (1MB-64KB) / 1MB≈93.75%;

[0065] Figure G=(V,E,W): V={v1, v2, ..., v 512} 512 nodes, E = {(i,j)|A ij >0} has a maximum of 16384 edges, and the weight W directly uses the value in A without additional storage;

[0066] Total memory requirements for the dynamic graph construction layer: Input H 0 =512×768×4 = 1.57 MB (from the input processing layer), similarity calculation (blocking) 2 KB (temporary), sparse matrix A is 64 KB, total runtime is approximately 1.6 MB.

[0067] Propagation Control Layer: The propagation control layer includes a seed node selection module, a seed query generation module, and a query propagation module. The propagation control layer receives the output of the dynamic graph construction layer (H... 0 (A, G), by selecting a seed node and propagating the Query, the transformation from full computation to sparse propagation is achieved, and the propagation uses message passing with a fixed number of steps;

[0068] The seed node selection module selects nodes from the set V of the Token graph structure G=(V,E,W) output by the dynamic graph construction layer. A set of seed nodes S, consisting of nodes, satisfies ,in, ρ is the number of seed nodes; ρ is the seed ratio parameter, ranging from [0.1, 0.3], and the data comes from the hyperparameter configuration; n is the length of the input sequence, which comes from the output of the input processing layer. This is a floor operation; these seed nodes will perform the full query computation, and other nodes will obtain an approximate query through propagation; Select(·,·) is a selection function that selects a subset of nodes according to a specific strategy. Strategy options include: uniform random sampling, or based on degree centrality, or based on the position of the node in the graph;

[0069] The initial hidden state H output by the seed query generation module from the input processing layer 0 In the middle, extract the row vector corresponding to the index of the seed node set S to form H. 0 S ;

[0070] Through formula Generate seed query matrix Q s W φ It is the Query projection matrix, which is generated through backpropagation after random initialization;

[0071] The Query propagation module uses a graph structure to propagate the Query vector from the seed node to all nodes, generating a complete Query matrix Q that satisfies Q = Propagate(Q s A, t), where the Propagate function iteratively propagates information through a graph structure, realizing message passing based on the adjacency matrix; Q s The output comes from the seed query generation module, and the adjacency matrix A comes from the output of the dynamic graph construction layer. t is the propagation step number, typically ranging from 2 to 5, and the data comes from the hyperparameter configuration. The value principle ensures that information can reach all nodes.

[0072] The following is a calculation example in this layer:

[0073] Set the calculation conditions (n=512, d=768, ρ=0.2, k=32, t=3);

[0074] Traditional transformer query generation is 512×768 2 ×4= 1207959552 FLOPs≈1.2 GFLOPs;

[0075] The seed query generated by the method of this invention is 102×768. 2 ×4 = 240,844,800 FLOPs≈0.24 GFLOPs; query propagation is 3×512×32×768×4 = 150,994,944 FLOPs≈0.15 GFLOPs, totaling 0.24+0.15=0.39 GFLOPs;

[0076] The method of this invention reduces the computational load by approximately 67.5% compared to the traditional method (1.2-0.39) / 1.2. At the same time, the traditional method requires calculating the query for all 512 tokens, while this method only needs to calculate for 102 seeds, saving approximately 80% of the query computation memory.

[0077] Computational optimization layer: Used for sparse projection of Key and Value, and computation of attention output. The computational optimization layer includes a sparse KV projection module and an attention computation module. This layer receives the output (Q, S) from the propagation control layer and the input H from the processing layer. 0 By using sparse KV projection and optimized attention computation, memory usage is significantly reduced;

[0078] The sparse KV projection module only projects the Key and Value values ​​for seed nodes, while non-seed nodes are not calculated or approximated, thus significantly reducing computation and memory usage; through the formula K... s = H 0 s ·W k With V s = H 0 s ·W v The Key matrix K of the seed node is calculated. s and Value matrix V s K s Used for attention score calculation, V s Used for attention-weighted summation; H 0 s Output H from the input processing layer 0 The row corresponding to the S index; W k It is the key projection matrix, W vIt is the Value projection matrix, and the data comes from random initialization and learning through backpropagation;

[0079] The attention calculation module uses the full query matrix Q and the sparse key matrix K. s Calculate the attention score, then compare it with the sparse Value matrix V. s The attention output matrix Attn is obtained by weighted summation and passed to the output integration layer. The calculation formula is Attn = Softmax(Q·K). s T / V s Where Q comes from the output of the propagation control layer; K s T Let Q·K be the transpose of the seed key. s T This is the attention score matrix, representing the attention score of each token for all seed nodes; Scaling factor to prevent excessively large dot product values ​​and stabilize the softmax gradient; softmax normalization function; V s Weighted summation;

[0080] The traditional Transformer has a memory requirement of O(n^2) for the attention matrix. 2 The memory requirements are adjusted so that the attention matrix occupies O(n×d) memory, and the key and value cache occupies O(|S|×d) memory. This method optimizes this by using sparse KV projection and attention calculation, resulting in a memory requirement of O(n×|S|) for the attention matrix and O(|S|×d) for the key and value cache. ρ is the seed ratio parameter, with a value range of [0.1, 0.3], which significantly reduces memory usage;

[0081] Since sparse KV projection only calculates the K and V of the seed node, attention calculation only requires an n×|S| fraction matrix, and the algorithm complexity of traditional transformers is O(n^2). 2 The algorithm complexity of the method of this invention is O(n·|S|) = O(n·ρ·n) = O(ρ·n) 2 ), memory saving ratio 1-ρ;

[0082] The following is a calculation example in this layer:

[0083] Set the calculation conditions (n=512, d=768, |S|=102);

[0084] In a traditional transformer, the K-projection is calculated as 512×768. 2 =302 MFLOPs, V projection calculated as 512×7682 =302 MFLOPs, Q·K T The calculation is 512×512×768=201 MFLOPs, the attention weight V is calculated as 512×512×768=201 MFLOPs, and the total is 1006 MFLOPs;

[0085] In the method of this invention, the K projection is calculated to be 102×768. 2 = 60 MFLOPs, V projection calculated as 102×768 2 =60 MFLOPs, Attention score matrix Q·K s T The calculation is 512×102×768=40 MFLOPs, the attention weight V is calculated as 512×102×768=40 MFLOPs, for a total of 200 MFLOPs; the computational load is reduced by (1006-200) / 1006≈80%.

[0086] Feedback adjustment layer: This layer includes an error detection module, a selective recalculation module, and a parameter adaptation module. This layer monitors the error propagated by the query, dynamically selects the nodes that need to be accurately recalculated, and adaptively adjusts the system parameters to ensure model accuracy.

[0087] The error detection module uses the L2 norm (Euclidean distance) to calculate the propagation Query(Q) and reference Query(Q). re The error vector between f) is formulated as ε = ||Q - Q re f||, where the error vector ε is a vector of dimension n (one error value for each token); ||·|| represents the L2 norm operation;

[0088] In the selective recalculation module, an error threshold θ is first set, which serves as the error judgment benchmark at the current time. Then, the error ε for each token is calculated. i Compared with θ, when the error of a node exceeds the threshold θ, it is determined that the propagation query accuracy of the token is substandard, using the formula Q. i =H 0 i ·W φ Perform precise query recomputation on this node, where H 0 i This represents the initial hidden state of the i-th token, with data derived from the i-th row of the input processing layer's output; W φ For the Query projection matrix, the data comes from the same parameters used in the propagation control layer; Q i This indicates the recalculated, exact query, replacing the original approximation.

[0089] The parameter adaptive module uses formulas Calculate the error threshold for the next time step, where θ t+1 The threshold representing the next time step, used for the next batch or layer; θ t λ represents the threshold at the current moment, and the initial value can be preset by hyperparameters; λ represents the learning rate, with a typical value of 0.01~0.1, and the data comes from the hyperparameter settings to control the adjustment step size. This represents the error gradient.

[0090] If the error increases ( If the error is greater than 0, then increase θ to reduce recalculation. If the error decreases ( If θ < 0, then reduce θ and increase recalculation. By dynamically updating the error threshold, accuracy and efficiency can be balanced.

[0091] The following is a calculation example in this layer:

[0092] Set the calculation conditions (n=512, d=768, |S|=102);

[0093] Assuming 20 nodes are detected that require recalculation (approximately 4%), the additional computational cost is:

[0094] Error detection: 512 × 768 = 0.4 MFLOPs (relatively small); Selective recalculation: 20 × 768 × 768 = 11.8 MFLOPs; Total additional cost approximately 12 MFLOPs;

[0095] The impact relative to the propagation control layer:

[0096] The propagation control layer query propagation is approximately 390 MFLOPs, and the feedback recalculation is 12 MFLOPs, an increase of approximately 3% (12 / 390). The additional overhead is small, but it can significantly improve accuracy.

[0097] Output integration layer: The output integration layer includes residual connection and layer normalization modules as well as feedforward network modules, and the output integration layer is compatible with traditional Transformer;

[0098] The residual connection and layer normalization module uses formula H 1 =LayerNorm(Attn+H) 0 The attention output is added to the original input via a residual connection, followed by layer normalization to maintain training stability and gradient flow; where Attn represents the attention, and the data is self-computed to optimize the layer's output; H 0 This represents the initial hidden state, with data coming from the output of the input processing layer. It provides residual connections to prevent information loss; LayerNorm(·) is used to independently normalize the d-dimensional vector of each token.

[0099] In the feedforward network module, the feedforward network FFN is constructed using the function FFN(x) = GELU(x·W1+b1)·W2+b2, where W1 is the weight of the expansion layer with a dimension of d×(4d); W2 is the weight of the compression layer with a dimension of (4d)×d; GELU is the activation function, making it a standard two-layer feedforward network, the same as the traditional Transformer; since the computational complexity of calculating the optimization layer Attn has been reduced from O(n... 2 d) decreased to O(n·|S|·d), and memory usage has decreased from O(n 2 The optimization efficiency is reduced to O(n·|S|). This layer only integrates these optimization results and is not the focus of optimization itself.

[0100] The following is a calculation example in this layer:

[0101] Set the calculation conditions (n=512, d=768);

[0102] FFN computational cost: First layer: 512 × 768 × 3072 = 1.2 GFLOPs; Second layer: 512 × 3072 × 768 = 1.2 GFLOPs; Total: 2.4 GFLOPs;

[0103] Overall system optimization effect: The traditional method requires attention (1 GFLOPs) + FFN (2.4 GFLOPs) = 3.4 GFLOPs; the method of this invention requires attention (0.2 GFLOPs) + FFN (2.4 GFLOPs) = 2.6 GFLOPs.

[0104] Savings: 23.5%

[0105] Memory analysis: When the system stacks N layers, the computational cost of the traditional transformer is N × 3.4 GFLOPs; the computational cost of this invention is N × 2.6 GFLOPs; the 12-layer model saves (3.4 GFLOPs - 2.6 GFLOPs) × 12 = 9.6 GFLOPs; therefore, for long sequences (such as n=2048), the traditional method cannot handle them (memory overflow), while this invention can run normally (memory reduction of 80%).

[0106] It also includes a parameter initialization module for initializing the projection matrix, seed ratio, sparsity coefficients, propagation steps, and error threshold. The following parameters need to be initialized upon system startup:

[0107] Model parameter initialization:

[0108] Query projection matrix W Q Xavier initialization, dimension d×d;

[0109] Key projection matrix W K Xavier initialization, dimension d×d;

[0110] Value projection matrix W V Xavier initialization, dimension d×d;

[0111] FFN weight matrices W1, W2: Xavier initialization;

[0112] Hyperparameter settings:

[0113] Seed ratio ρ: Default value 0.2, adjustable range [0.1, 0.3];

[0114] Sparsity coefficient α: Default value 10, used for calculation ;

[0115] Propagation steps t: Default value 3, adjustable range [2, 5];

[0116] Error threshold θ: Initial value 0.3, subsequently adjusted adaptively;

[0117] Learning rate λ: Default value 0.01, used for updating θ.

[0118] In this invention, gradient calculation is achieved through automatic differentiation, and all parameters can be updated through standard backpropagation; the projection matrix W Q W K W V The gradient of the FFN parameter; the gradient of the learnable graph construction parameters (e.g., using a learnable similarity function);

[0119] In special scenarios, such as processing very short sequences (n < 50), setting ρ = 0.5 (increasing the seed ratio) can degenerate into standard attention; for processing very long sequences (n > 10000), the sequence is divided into multiple segments, and information is passed between segments through overlap or global tokens. When memory is insufficient, the degradation is to automatically increase sparsity (decrease k) and reduce the number of seeds (decrease ρ).

[0120] A method for optimizing a Transformer system based on a token graph-based query propagation mechanism includes the following steps:

[0121] S1. Input Encoding Stage: Token encoding and position encoding are performed on the input sequence to generate the initial hidden state;

[0122] S2. Graph Construction Stage: Based on the initial hidden state, calculate the cosine similarity or scaled dot product as the similarity value to form a similarity matrix between tokens. Then, according to the number of the most important neighbors retained by each node, perform matrix sparsification to generate a sparse adjacency matrix. At the same time, define the node set, edge set, and weight function based on the sparse adjacency matrix to construct the token graph.

[0123] S3, Seed Selection Stage: Select a set of seed nodes from the Token graph. The selection strategy includes uniform random sampling, degree centrality-based selection, or node position-based selection. And generate a complete query vector by linearly projecting only the initial hidden state of the seed nodes.

[0124] S4, Query Propagation Phase: The Query vector of the seed node is iteratively propagated through the sparse adjacency matrix using the propagation function to generate the Query matrix of the entire sequence. The number of propagation steps is a preset value to ensure full graph coverage.

[0125] S5, KV projection stage: Key and Value matrices are generated only for the initial hidden state projection of the seed nodes; projection is omitted for non-seed nodes.

[0126] S6. Attention Calculation Stage: Calculate the dot product of the Query matrix and the seed Key matrix, apply softmax normalization, and then sum the product with the seed Value matrix using weighted summation. Perform attention calculation and generate attention output.

[0127] S7. Feedback Control Phase: Calculate the norm error between the propagated Query and the reference Query; recalculate the accurate Query for nodes where the error exceeds the threshold; adaptively update the threshold based on the error gradient.

[0128] It should be noted that the above embodiments are merely illustrative of several implementation methods of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be pointed out that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A Transformer optimization system based on a Token Graph-based Query Propagation Mechanism, characterized in that, include: Input processing layer: used to encode the input sequence to generate the initial hidden state; Dynamic graph construction layer: used to construct the token graph and sparse adjacency matrix based on the initial hidden state mentioned above; Propagation control layer: used to select seed nodes, generate seed queries, and propagate the full query matrix through the token graph; Computational optimization layer: used for sparse projection seeds (Key and Value), and computes attention output; Feedback adjustment layer: used to detect propagation errors, selectively recalculate, and adaptively adjust parameters; Output integration layer: used for residual connections, layer normalization, and feedforward network processing to generate the final output.

2. The Transformer optimization system based on the Token Graph Query Propagation Mechanism according to claim 1, characterized in that: The input data in the input processing layer comes from the output of the word segmenter.

3. The Transformer optimization system based on the Token Graph Query Propagation Mechanism according to claim 1, characterized in that: The dynamic graph construction layer includes a similarity calculation module, a matrix sparsification module, and a graph generation module. In the matrix sparsification module, sparsification is based on the number of neighbors retained for each node, which is related to the logarithm of the sequence length.

4. The Transformer optimization system based on the Token Graph Query Propagation Mechanism according to claim 1, characterized in that: The propagation control layer includes a seed node selection module, a seed query generation module, and a query propagation module. The propagation control layer receives the output of the dynamic graph construction layer and achieves the transformation from full computation to sparse propagation by selecting seed nodes and propagating queries. The propagation uses message passing with a fixed number of steps.

5. The Transformer optimization system based on the Token Graph Query Propagation Mechanism according to claim 1, characterized in that: The computation optimization layer includes a sparse KV projection module and an attention computation module. The computational complexity of the attention is O(n × S × d), where S is the number of seeds.

6. The Transformer optimization system based on the Token Graph Query Propagation Mechanism according to claim 1, characterized in that: The feedback adjustment layer includes an error detection module, a selective recalculation module, and a parameter adaptation module, wherein the error is calculated using the L2 norm.

7. The Transformer optimization system based on the Token Graph Query Propagation Mechanism according to claim 1, characterized in that: The output integration layer includes a residual connection and layer normalization module and a feedforward network module, and the output integration layer is compatible with the traditional Transformer.

8. The Transformer optimization system based on the Token Graph Query Propagation Mechanism according to claim 1, characterized in that: It also includes a parameter initialization module for initializing the projection matrix, seed ratio, sparsity coefficient, propagation steps, and error threshold.

9. A method for optimizing a Transformer system based on a Token Graph-based Query Propagation Mechanism according to claim 1, characterized in that, Includes the following steps: S1. Input Encoding Stage: Token encoding and position encoding are performed on the input sequence to generate the initial hidden state; S2. Graph Construction Stage: Based on the initial hidden state, calculate the cosine similarity or scaled dot product as the similarity value to form a similarity matrix between tokens. Then, according to the number of the most important neighbors retained by each node, perform matrix sparsification to generate a sparse adjacency matrix. At the same time, define the node set, edge set, and weight function based on the sparse adjacency matrix to construct the token graph. S3, Seed Selection Phase: Select a set of seed nodes from the Token graph, and generate a complete Query vector by linearly projecting only the initial hidden state of the seed nodes; S4, Query Propagation Phase: The Query vector of the seed node is iteratively propagated through the sparse adjacency matrix using the propagation function to generate the Query matrix of the entire sequence. The number of propagation steps is a preset value to ensure full graph coverage. S5, KV projection stage: Key and Value matrices are generated only for the initial hidden state projection of the seed nodes; projection is omitted for non-seed nodes. S6. Attention Calculation Stage: Calculate the dot product of the Query matrix and the seed Key matrix, apply softmax normalization, and then sum the product with the seed Value matrix using weighted summation. Perform attention calculation and generate attention output. S7. Feedback Control Phase: Calculate the norm error between the propagated Query and the reference Query; recalculate the accurate Query for nodes where the error exceeds the threshold; adaptively update the threshold based on the error gradient.

Images