A language model structure suitable for ultra-long text

Abstract

The application discloses a language model structure suitable for super-long text, comprising: after receiving a multi-modal data stream, an asynchronous multi-modal fusion layer performs cross-modal feature fusion through a cross-modal attention gate unit when semantic correlation exceeds a threshold to obtain a fusion input feature; a recurrent state space unit processes the fusion input feature to obtain a state vector; a hybrid expert attention module is used to combine a global sparse attention mechanism and a local window attention mechanism, and an attention routing network is used to select an optimal attention calculation mode for different Token features; a routing prediction network of an MoE dynamic routing layer predicts the load demand of subsequent Token according to the Token feature distribution in the current window, adjusts the routing weight of the dynamic gate network, and uniformly distributes the calculation task to multiple expert networks. The application is efficient in calculation, economical in memory, has long-term memory capacity, is balanced in load, and can accurately process multi-modal information.

Description

Technical Field

[0001] This invention relates to the field of language models, and in particular to a language model structure suitable for very long texts. Background Technology

[0002] In recent years, large language models have demonstrated powerful capabilities in tasks such as natural language processing, multimodal understanding, and generation, becoming a crucial engine driving the development of artificial intelligence. However, with the continuous expansion of model parameter scale, the dramatic increase in the demand for context length in application tasks, and the increasing complexity of multimodal scenarios, the existing basic model architecture centered on the standard Transformer and its accompanying training and inference paradigms are facing a series of interconnected and mutually restrictive fundamental technical bottlenecks. These bottlenecks are mainly manifested in the following ways: when pursuing larger-scale, longer sequences, and more complex tasks, the existing technical system suffers from systemic defects in computational efficiency, memory efficiency, and information processing performance, severely limiting the scalability, economy, and upper limit of capabilities of the models.

[0003] Specifically, firstly, regarding model training and scaling, the linear scaling law widely relied upon in the industry fails at extremely large scales, leading to severe misplanning of training resources for models with hundreds of billions of parameters. This results in either enormous waste of computing power or underfitting due to insufficient resources, severely undermining the feasibility of efficiently scaling the model according to the expected path. Secondly, in terms of the efficiency of the core model architecture, the complexity of the self-attention mechanism increases quadratically with the sequence length. This makes it unacceptable for the model to handle extremely long contexts with more than 100,000 or even millions of tokens, both inference speed and memory usage become unacceptable, greatly limiting the model's application in key scenarios such as long document understanding and long video analysis. More fundamentally, the current structure lacks an explicit memory mechanism that can store historical information losslessly or efficiently, failing to achieve true infinite context processing and continuous learning capabilities, and possessing an inherent "information forgetting" boundary.

[0004] Furthermore, problems are equally prominent in cutting-edge areas such as extended architectures aimed at improving model capacity and multimodal fusion. For example, static and short-sighted routing strategies in hybrid expert architectures easily lead to severe uneven distribution of expert load, resulting in a significant decrease in the utilization of computational resources in distributed training, deviating from the original intention of improving efficiency through sparsity. Meanwhile, when processing multimodal information with different temporal granularities, such as video, audio, and text, existing methods typically employ simplistic and crude forced alignment strategies. This not only introduces a large amount of redundant computation but may also destroy the temporal structure and high-frequency information of the original data, leading to poor fusion results. These specific problems collectively reveal the deep-seated shortcomings of existing technological paradigms in terms of scalability, adaptability, and overall performance. Therefore, developing a new generation of model architectures and technological systems that can fundamentally overcome these efficiency and capability bottlenecks has become a crucial issue that urgently needs to be addressed to propel large models towards higher levels of intelligence. Summary of the Invention

[0005] To address the aforementioned issues, the present invention aims to provide a language model structure suitable for extremely long texts that is computationally efficient, memory-efficient, possesses long-range memory capabilities, has load balancing, and can accurately process multimodal information.

[0006] Based on this, the present invention provides a language model structure suitable for ultra-long texts, the structure comprising: The asynchronous multimodal fusion layer is used to receive multimodal data streams and perform cross-modal feature fusion when the semantic relevance exceeds a preset threshold through a learnable cross-modal attention gating unit to obtain fused input features; A cyclic state space unit is used to receive the fused input features and process the fused input features using a preset linear recursive formula to obtain the state vector at the current time. The hybrid expert attention module is used to receive the state vector after it has been processed by the cyclic state space unit, and combine the global sparse attention head and the local sliding window attention head to select the optimal attention calculation mode for tokens with different features through the attention router. The MoE dynamic routing layer includes a route prediction network and a dynamic gating network. The route prediction network predicts the load demand of subsequent tokens based on the token feature distribution in the current window, and dynamically adjusts the routing weights of the dynamic gating network based on the load demand, so as to evenly distribute the computing tasks to multiple expert networks.

[0007] The multimodal data stream includes: text stream, video stream, and audio stream.

[0008] The asynchronous multimodal fusion layer includes: multiple independent modal encoders, a time alignment buffer pool, and a cross-modal attention gating unit; The independent modal encoder extracts features from the video stream, audio stream, and text stream respectively, and obtains independent features for each modality, namely text features, video features, and audio features. The time alignment buffer is used to temporarily store the modal-independent features; The cross-modal attention gating unit is used to calculate the mutual attention score between the text features, the video features, and the audio features. When the mutual attention score is higher than the fusion threshold, feature splicing and projection operations are performed based on the independent features of each modality to obtain the fused input features.

[0009] The linear recursive formula includes: ; Wherein, A is the diagonalized state transition matrix, B is the input projection matrix, and h t h is the state vector at the current moment. t-1 Let x be the state vector of the previous time step. t The fused input feature is the current input.

[0010] The cyclic state space unit supports parallel training mode and recursive inference mode. In the parallel training mode, recursive computation is transformed into parallel convolution operation with a complexity of O(LlogL). In the recursive inference mode, only the state vector of the previous time step is read to complete the computation with a complexity of O(1).

[0011] The hybrid expert attention module includes: The system includes a global sparse attention head, a local sliding window attention head, and an attention router. The global sparse attention head is used to extract key token features with high information entropy from a sequence. The local sliding window attention head is used to extract features of neighboring tokens within a preset window before and after the current token; the attention router calculates the matching degree between the query vector of the input token and different attention modes, outputs the mode selection probability, and determines the optimal attention calculation mode based on the mode selection probability.

[0012] The hybrid expert attention module is also equipped with a KV Cache management unit; the KV Cache management unit discards expired KV Cache for the local sliding window attention head, and retains only the KV Cache of key tokens for the global sparse attention head.

[0013] The routing prediction network is a lightweight multilayer perceptron. The routing prediction network takes the mean of the token embedding of the current sliding window as input and outputs the predicted load distribution of the future K tokens on each expert network. The dynamic gating network adds a load balancing bias term under the Top-K selection rule based on the predicted load distribution, corrects the routing probability and evenly distributes the computation tasks.

[0014] The MoE dynamic routing layer is configured with a load balancing loss calculation unit. The load balancing loss calculation unit is used to calculate the difference between the token processing ratio and the average routing probability of each expert network, and generate a load balancing auxiliary loss to constrain the load uniformity of each expert network.

[0015] The cyclic state space unit, the hybrid expert attention module, and the MoE dynamic routing layer are integrated within the Transformer Block.

[0016] By employing this invention, the following technical effects can be achieved: By combining recursive structures with infinite state compression capabilities (such as RSSU) with efficient long sequence computation models (such as MoA), a new generation of core architecture has been constructed. This architecture not only theoretically supports infinitely long context processing, but also significantly reduces the computational and memory complexity of processing ultra-long sequences from the quadratic level of traditional Transformers to linear or near-linear levels in practice. This enables the model to achieve high-precision understanding and generation of documents and videos with millions of tokens at an acceptable cost and low latency, completely solving the problem of the infeasibility of long context applications caused by memory explosion and computational bottlenecks.

[0017] To address the inherent load imbalance challenge in hybrid expert architectures, an intelligent dynamic routing strategy is designed to adaptively adjust expert activation based on real-time load and data distribution. This effectively eliminates the phenomenon of overloaded "hot experts" and idle "unpopular experts," resulting in a more balanced load across computing units (such as GPUs) in large-scale distributed training clusters. This significantly improves hardware utilization and overall training throughput, ensuring the actual efficiency of sparse large model expansion paths.

[0018] This invention abandons the crude method of forcibly projecting different modalities onto the same fixed-rate sequence in existing technologies, and innovatively adopts an asynchronous fusion mechanism. This mechanism respects and preserves the original, distinct temporal granularity and temporal characteristics of modalities such as video, audio, and text, avoiding the loss of high-frequency information and temporal misalignment caused by resampling. Thus, it achieves an optimal balance between computational efficiency and information fidelity, ultimately significantly improving the model's understanding accuracy and generation quality in complex cross-modal tasks (such as video understanding and audio subtitle generation).

[0019] In summary, the technical effects of this invention are systematic and synergistic: it aims to deliver a new generation of large-scale model foundational framework that combines high efficiency in processing ultra-long sequences, balanced distributed computing, and accurate multimodal understanding. This framework is expected to fundamentally overcome the core bottlenecks of existing architectural paradigms when moving towards larger-scale and more complex tasks, providing crucial infrastructure support for realizing more powerful, practical, and cost-effective general-purpose artificial intelligence systems. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a language model structure suitable for ultra-long texts provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the working principle and load balancing process of the MoE dynamic routing layer provided in this embodiment of the invention. Figure 3 This is a flowchart of model training provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Figure 1 This is a schematic diagram of a language model structure suitable for very long texts provided in an embodiment of the present invention. The structure includes: The asynchronous multimodal fusion layer 101 is used to receive multimodal data streams and perform cross-modal feature fusion when the semantic relevance exceeds a preset threshold through a learnable cross-modal attention gating unit to obtain fused input features; The cyclic state space unit 102 is used to receive the fused input features and process the fused input features using a preset linear recursive formula to obtain the state vector at the current time. The hybrid expert attention module 103 is used to receive the state vector after being processed by the cyclic state space unit, and combine the global sparse attention mechanism and the local window attention mechanism to select the optimal attention calculation mode for tokens with different features through the attention routing network. Figure 2 This is a schematic diagram of the working principle and load balancing process of the MoE dynamic routing layer provided in this embodiment of the invention. The MoE dynamic routing layer 104 includes a route prediction network and a dynamic gating network. The route prediction network predicts the load demand of subsequent tokens based on the token feature distribution in the current window, and dynamically adjusts the routing weight of the dynamic gating network based on the load demand to evenly distribute the computing tasks to multiple expert networks.

[0024] Among them, the asynchronous multimodal fusion layer 101 is located at the input end of the model. It is designed with an independent modal encoder and a cross-modal attention gating network unit, which allows different modalities to be input at the original rate and performs sparse fusion only in semantic keyframes, thus solving the redundancy and distortion problems caused by sampling rate mismatch.

[0025] The multimodal data stream includes: text stream, video stream, and audio stream.

[0026] The asynchronous multimodal fusion layer includes: multiple independent modal encoders, a time alignment buffer pool, and a cross-modal attention gating unit; The independent modal encoder extracts features from the video stream, audio stream, and text stream respectively, obtaining independent features for each modality, namely text features, video features, and audio features. Specifically, the text encoder extracts text word embeddings and positional features; the video encoder extracts spatiotemporal visual features based on existing 3D-CNN or ViT (Visual Transformer) architectures; and the audio encoder extracts acoustic features based on one-dimensional convolution of the audio Mel spectrogram.

[0027] The time alignment buffer pool is used to temporarily store the independent features of each modality. The time alignment buffer pool is not only used for data caching, but also includes a timestamp index module, which is used to map video streams (such as 24fps) and audio streams (such as 16kHz) with different native sampling rates to the corresponding sliding windows according to the absolute timestamp, so as to achieve temporal alignment and standby.

[0028] The cross-modal attention gating unit is used to calculate the mutual attention score between the text features, the video features, and the audio features. When the mutual attention score is higher than the fusion threshold, feature splicing and projection operations are performed based on the independent features of each modality to obtain the fused input features.

[0029] The process of obtaining the mutual attention score is existing technology. As an optional embodiment, the calculation of the mutual attention score is based on the standard cross-attention mechanism: a query vector is generated from the text features through linear mapping, a key vector is generated from the video / audio features in the buffer pool, the dot product of the query vector and the key vector is calculated and scaled, and finally the mutual attention score is obtained by normalization through the Softmax function.

[0030] The feature concatenation and projection operation is as follows: when the mutual attention score is higher than the fusion threshold, select the video features and / or audio features corresponding to the score, and concatenate them with the current text features in the hidden dimension; then, input the concatenated high-dimensional vector into a feature projection layer composed of a multilayer perceptron (MLP), reduce the dimension and map it back to the unified feature dimension of the base model, and output the fused input features.

[0031] During the independent encoding phase, each modality employs an encoder adapted to its characteristics to preserve the original temporal structure: text is encoded using Word Embedding, and video is encoded frame-by-frame using 3D-CNN or ViT. Feature extraction is performed on audio data using Mel spectrograms and Conv1D based on time delay. Encoding is completed; then, in the asynchronous gating stage, text features are used as queries, and the timing of fusion is dynamically determined through a cross-attention mechanism. The gating unit G selects audio and video features within the corresponding time window based on the text timestamp τ to achieve temporal alignment and correlation evaluation between modalities; finally, in sparse fusion, a threshold θ is set, and audio and video features are projected onto the text space and superimposed only when the attention score exceeds the threshold (such as when the text clearly points to audiovisual content or there is modal conflict). Otherwise, the input remains pure text, thereby effectively avoiding continuous dense fusion calculations, reducing noise interference, and preserving the original sampling rate and independence of each modality.

[0032] The linear recursive formula includes: ; Wherein, A is the diagonalized state transition matrix, B is the input projection matrix, and h t h is the state vector at the current moment. t-1 Let x be the state vector of the previous time step. t The fused input feature is the current input.

[0033] The diagonalized state transition matrix A is a set of learnable parameters used to control the decay ratio of historical state information from the previous time step and the degree of long-range memory retention; the input projection matrix B is a parameterized matrix used to calculate the contribution weight of the current input features to the internal state update.

[0034] The cyclic state space unit supports both parallel training and recursive inference modes; in the parallel training mode, recursive computation is transformed into parallel convolution operations. ,in, The SSM core has a complexity of O(LlogL); in the recursive reasoning mode, the recursive formula is used directly. (where Ā and These are the parameter matrices obtained by discretizing the basic state transition matrix A and the input projection matrix B using step size processing, respectively. This is achieved by using the current input feature x... t Multiplying with matrix B completes the input projection operation on the fused input features. Each inference only requires reading the state h from the previous time step. t-1 The calculation is completed by reading only the state vector of the previous time step, with a complexity of O(1).

[0035] Where L represents the sequence length of the input data. O(LlogL) represents the computational complexity of performing global convolution through Fast Fourier Transform (FFT) in parallel training mode; O(1) represents that the computational cost required to generate each new token in recursive inference mode is constant and does not increase with the increase of the context length L.

[0036] The hybrid expert attention module includes: The system comprises a global sparse attention head, a local sliding window attention head, and an attention router. The global sparse attention head is used to extract key token features with high information entropy in the sequence. The local sliding window attention head is used to extract features of neighboring tokens within a preset window before and after the current token. The attention router calculates the matching degree between the query vector of the input token and different attention modes, outputs the mode selection probability, and determines the optimal attention calculation mode based on the mode selection probability.

[0037] The sequence refers to the multidimensional feature vector sequence processed by the asynchronous multimodal fusion layer and the cyclic state space unit; the high information entropy key token feature is selected by introducing a learnable threshold parameter during the training process to filter out feature vectors that have a strong causal relationship or are rare in predicting subsequent text.

[0038] The input token is a state vector processed by RSSU (Recurrent State Space Unit). The attention router receives the input token through a normalized linear classification network and outputs a two-dimensional matching degree vector (Logits). It then uses a Softmax activation function to transform the Logits into mode selection probabilities assigned to the global sparse head and the local sliding window head. The system employs a greedy selection (Top-1) or Gumbel-Softmax sampling strategy to determine the highest probability as the optimal attention calculation mode.

[0039] After obtaining the optimal attention computation mode, the token will only be distributed to the corresponding attention head (global sparse or local sliding window) for feature extraction computation. This avoids fully dense self-attention computation while preserving key context, significantly reducing computational overhead. This mode does not interfere with subsequent MoE routing distribution.

[0040] The hybrid expert attention module also includes an attention aggregation unit, which is used to perform weighted summation or splicing projection on the feature sequences output by different attention heads (expert computing modes) according to the probability selection of the mode, to complete feature aggregation.

[0041] The hybrid expert attention module is equipped with a KV Cache management unit. This unit discards expired KV Cache entries for the local sliding window attention head and retains only the KV Cache entries for the global sparse attention head. This reduces the memory usage for inference with millions of tokens to less than 10% of standard attention.

[0042] The routing prediction network is a lightweight multilayer perceptron. The routing prediction network takes the mean of the token embedding of the current sliding window as input and outputs the predicted load distribution of the future K tokens on each expert network. The dynamic gating network adds a load balancing bias term under the Top-K selection rule based on the predicted load distribution, corrects the routing probability and evenly distributes the computation tasks.

[0043] The KV Cache management unit allocates a fixed-length (e.g., 1024) ring buffer for local sliding windows. When a token is written outside the window, the oldest feature cache is overwritten using the first-in-first-out (FIFO) principle to discard expired tokens. For global sparse attention tokens, an independent persistent storage area is allocated, and only the features of key tokens are appended and not discarded, thus keeping the memory usage constant during long text inference.

[0044] The routing prediction network introduces a miniature Transformer or MLP predictor before the standard Gating Network. It reads the current token and its preceding context to predict the domain distribution (e.g., code, mathematics, literature) of the next K tokens.

[0045] This invention calculates S = xW in the standard gating system. g Based on this, a dynamic bias b is introduced. dyn The calculation formula is as follows: S' = xW g + b dyn ; Where S' is the corrected gated routing score, x is the feature vector of the current input token, Wg is the weight matrix of the gated network, and b dyn This is the bias value dynamically generated by the routing prediction network for the current expert.

[0046] This bias is dynamically generated by the predictor. When the predictor determines that a large influx of "code" type tokens will occur in the future, potentially overloading "code experts," it will decrease the bias value b of the corresponding expert. dyn,code This directs traffic to backup specialists with similar capabilities, enabling proactive load regulation.

[0047] At the same time, a load balancing loss term L is introduced into the training loss. load : ; Among them, the The proportion of tokens processed by expert i, the Let N represent the average probability of routing to expert i, and let N represent the total number of experts.

[0048] This loss term helps the routing network maintain a uniform load on each expert in the long run, thus avoiding the problem of uneven utilization of experts.

[0049] The MoE dynamic routing layer is configured with a load balancing loss calculation unit. The load balancing loss calculation unit is used to calculate the difference between the token processing ratio and the average routing probability of each expert network, and generate a load balancing auxiliary loss to constrain the load uniformity of each expert network.

[0050] The routing prediction network receives the hidden layer embedding representation of the token within the current sliding window, performs feature dimensionality reduction and nonlinear activation through a multilayer perceptron (MLP), and outputs a prediction vector. After being normalized by Softmax, this prediction vector represents the expected probability that the input tokens in the next K time steps will be allocated to various expert networks (such as code experts, logic experts, etc.), which serves as the future load requirement.

[0051] The cyclic state space unit, the hybrid expert attention module, and the MoE dynamic routing layer are integrated within the Transformer Block.

[0052] The overall training process integrates the above technologies, and the specific steps are as follows: Initialization: Initialize the model parameters based on the resource planning results.

[0053] Data stream input: Video, audio, and text data streams are input into the model after passing through the asynchronous fusion layer.

[0054] Before formally building and training the model, this invention first performs a resource planning step. To address the biases inherent in traditional linear scaling laws at extremely large scales, this embodiment employs the following nonlinear scaling law model for prediction: ; Where K is the size of the look-ahead prediction window (e.g., predicting K=64 tokens in the future). E represents the irreducible minimum error due to data quality constraints, A and B are the performance improvement coefficients brought by the number of parameters and the amount of data, respectively, α and β are the corresponding decay exponents, λ is the weight coefficient of the nonlinear correction term, k is the sigmoid function slope parameter, γ is the optimal data / parameter ratio constant for model training (e.g., 20), and L(C) represents the expected loss under the computational budget C. Where D is the number of model parameters and D is the number of training tokens. The last term in the formula is a non-linear correction term, used to describe the performance saturation effect caused when the data-parameter ratio D / N deviates from the optimal range.

[0055] The specific implementation steps are as follows: 1. Small-scale probing: Within 1% of the budget, train a series of small models with different combinations of N and D (such as 100M, 500M, 1B parameter count) and obtain the corresponding loss values.

[0056] 2. Parameter fitting: Based on the loss data of the small model, the constant terms A, B, α, β, λ, k, γ in the above formula are fitted using the least squares method.

[0057] 3. Reverse solution: Set the total calculation budget C_ total ≈ 6ND, using the Lagrange multiplier method to find the optimal number of parameters N* and the optimal number of training tokens D* that minimize L(C).

[0058] 4. Optimal output: Outputs the optimal number of model layers, hidden layer dimensions, and the amount of training data required, providing guidance for subsequent model architecture construction.

[0059] After resource planning is completed, the model training process begins. Figure 3 This is a flowchart of model training provided in an embodiment of the present invention, and the specific steps are as follows: Data flows through the RSSU module, where historical information is compressed into state h; Upon entering the MoA module, the router assigns an attention computation path to each token; When using the MoE module, the predictor anticipates the load and dynamically adjusts the routing, distributing tokens to experts on different GPUs for computation.

[0060] Backpropagation and parameter update: Calculate the total loss L total = L task + αL load + βL aux , where L task L represents the model's base loss on the actual task, i.e., the task principal loss. load This represents the load balancing loss term, also known as expert load balancing loss, where α represents the weighting coefficient of the load balancing loss, and L... aux This represents the auxiliary supervision loss or routing auxiliary loss, where β represents the auxiliary loss weighting coefficient, and L... task αL is used to ensure the performance of the model task. load To ensure load balancing for MoE experts and fully utilize multi-GPU resources, βL aux To assist in stable training, the routing and state modules are optimized. The three factors are weighted and summed to achieve a balance between task accuracy, load balancing, and training stability.

[0061] Perform gradient backpropagation to update model parameters, SSM state parameters, and parameters of each routing network.

[0062] By establishing and applying a nonlinear scaling law that more closely aligns with the actual model performance growth curve, computational resources during training can be precisely planned and controlled. This effectively avoids the blind investment and huge waste of computational resources caused by relying on traditional linear laws in training models with hundreds of billions or more parameters. Thus, it maximizes model performance under the same budget constraints, or significantly reduces training costs under predetermined performance targets, fundamentally improving the economy and return on investment of large-scale model training.

[0063] By employing this invention, the following technical effects can be achieved: By combining recursive structures with infinite state compression capabilities (such as RSSU) with efficient long sequence computation models (such as MoA), a new generation of core architecture has been constructed. This architecture not only theoretically supports infinitely long context processing, but also significantly reduces the computational and memory complexity of processing ultra-long sequences from the quadratic level of traditional Transformers to linear or near-linear levels in practice. This enables the model to achieve high-precision understanding and generation of documents and videos with millions of tokens at an acceptable cost and low latency, completely solving the problem of the infeasibility of long context applications caused by memory explosion and computational bottlenecks.

[0064] To address the inherent load imbalance challenge in hybrid expert architectures, an intelligent dynamic routing strategy is designed to adaptively adjust expert activation based on real-time load and data distribution. This effectively eliminates the phenomenon of overloaded "hot experts" and idle "unpopular experts," resulting in a more balanced load across computing units (such as GPUs) in large-scale distributed training clusters. This significantly improves hardware utilization and overall training throughput, ensuring the actual efficiency of sparse large model expansion paths.

[0065] This invention abandons the crude method of forcibly projecting different modalities onto the same fixed-rate sequence in existing technologies, and innovatively adopts an asynchronous fusion mechanism. This mechanism respects and preserves the original, distinct temporal granularity and temporal characteristics of modalities such as video, audio, and text, avoiding the loss of high-frequency information and temporal misalignment caused by resampling. Thus, it achieves an optimal balance between computational efficiency and information fidelity, ultimately significantly improving the model's understanding accuracy and generation quality in complex cross-modal tasks (such as video understanding and audio subtitle generation).

[0066] In summary, the technical effects of this invention are systematic and synergistic: it aims to deliver a new generation of large-scale model foundational framework that combines high efficiency in processing ultra-long sequences, balanced distributed computing, and accurate multimodal understanding. This framework is expected to fundamentally overcome the core bottlenecks of existing architectural paradigms when moving towards larger-scale and more complex tasks, providing crucial infrastructure support for realizing more powerful, practical, and cost-effective general-purpose artificial intelligence systems.

[0067] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention.

Claims

1. A language model structure suitable for extremely long texts, characterized in that, include: The asynchronous multimodal fusion layer is used to receive multimodal data streams and perform cross-modal feature fusion when the semantic relevance exceeds a preset threshold through a learnable cross-modal attention gating unit to obtain fused input features; A cyclic state space unit is used to receive the fused input features and process the fused input features using a preset linear recursive formula to obtain the state vector at the current time. The hybrid expert attention module is used to receive the state vector after it has been processed by the cyclic state space unit, and combine the global sparse attention head and the local sliding window attention head to select the optimal attention calculation mode for tokens with different features through the attention router. The MoE dynamic routing layer includes a route prediction network and a dynamic gating network. The route prediction network predicts the load demand of subsequent tokens based on the token feature distribution in the current window, and dynamically adjusts the routing weights of the dynamic gating network based on the load demand, so as to evenly distribute the computing tasks to multiple expert networks.

2. The structure according to claim 1, characterized in that, The multimodal data stream includes: text stream, video stream, and audio stream.

3. The structure according to claim 2, characterized in that, The asynchronous multimodal fusion layer includes: multiple independent modal encoders, a time alignment buffer pool, and a cross-modal attention gating unit; The independent modal encoder extracts features from the video stream, audio stream, and text stream respectively, and obtains independent features for each modality, namely text features, video features, and audio features. The time alignment buffer is used to temporarily store the modal-independent features; The cross-modal attention gating unit is used to calculate the mutual attention score between the text features, the video features, and the audio features. When the mutual attention score is higher than the fusion threshold, feature splicing and projection operations are performed based on the independent features of each modality to obtain the fused input features.

4. The structure according to claim 1, characterized in that, The linear recursive formula include: h t = A・h t-1 + B・x t ; Wherein, A is the diagonalized state transition matrix, B is the input projection matrix, and h t h is the state vector at the current moment. t-1 Let x be the state vector of the previous time step. t The fused input feature is the current input.

5. The structure according to claim 1, characterized in that, The cyclic state space unit supports parallel training mode and recursive inference mode. In the parallel training mode, recursive computation is transformed into parallel convolution operation with a complexity of O(LlogL). In the recursive inference mode, only the state vector of the previous time step is read to complete the computation with a complexity of O(1).

6. The structure according to claim 1, characterized in that, The hybrid expert attention module includes: The system includes a global sparse attention head, a local sliding window attention head, and an attention router. The global sparse attention head is used to extract key token features with high information entropy from a sequence. The local sliding window attention head is used to extract features of neighboring tokens within a preset window before and after the current token; the attention router calculates the matching degree between the query vector of the input token and different attention modes, outputs the mode selection probability, and determines the optimal attention calculation mode based on the mode selection probability.

7. The structure according to claim 6, characterized in that, The hybrid expert attention module is also equipped with a KVCache management unit; the KVCache management unit discards expired KVCache for the local sliding window attention head, and retains only the KVCache of key tokens for the global sparse attention head.

8. The structure according to claim 1, characterized in that, The routing prediction network is a lightweight multilayer perceptron; the routing prediction network takes the mean of the token embedding of the current sliding window as input and outputs the predicted load distribution of the future K tokens on each expert network; the dynamic gating network adds a load balancing bias term under the Top-K selection rule based on the predicted load distribution, corrects the routing probability and evenly distributes the computation tasks.

9. The structure according to claim 1, characterized in that, The MoE dynamic routing layer is configured with a load balancing loss calculation unit; the load balancing loss calculation unit is used to calculate the difference between the token processing ratio and the average routing probability of each expert network, and generate load balancing auxiliary loss to constrain the load uniformity of each expert network.

10. The structure according to claim 1, characterized in that, The cyclic state space unit, the hybrid expert attention module, and the MoE dynamic routing layer are integrated within the Transformer Block.

Images