Natural language processing method based on self-adaptive double-memory hybrid architecture

Abstract

The invention provides a natural language processing method based on a self-adaptive double-memory hybrid architecture, which comprises the following steps of: embedding conversion: acquiring an input text lexical element sequence, converting the input text lexical element sequence into embedding representation through an embedding layer, and providing input for a self-adaptive double-memory hybrid architecture model; complexity evaluation: based on the embedded representation, a sequence complexity evaluator of a dynamic memory layering mechanism is adopted; memory allocation; sequence segmentation; performing short-term processing; middle-stage treatment; long-term treatment; bidirectional fusion; performing forward optimization; outputting a task; precise modeling is kept in a key short-term window, an efficient mechanism is adopted in a medium-and-long-term window, optimal balance of efficiency and performance is achieved, a three-layer memory system is adopted, active information is captured in a short term, semantic association is maintained in a medium term, historical knowledge is compressed for a long term, multi-scale memory organic unification, two-way feedback and self-adaptive fusion are achieved, and the method is suitable for large-scale application. Different mechanisms are complementary to each other and are dynamically coordinated, and optimal configuration is automatically achieved in different task scenes.

Description

Technical Field

[0001] This invention relates to a natural language processing method based on an adaptive dual-memory hybrid architecture, belonging to the field of natural language processing technology. Background Technology

[0002] Natural language processing is a core direction in the field of artificial intelligence, aiming to enable computers to understand, generate and process human language. Since the introduction of the Transformer architecture, models based on the self-attention mechanism have made breakthroughs in tasks such as machine translation and text generation, becoming the mainstream paradigm in this field. However, traditional Transformers suffer from a fundamental complexity bottleneck: the time and space complexity of standard self-attention is both 1 / 3. In long text processing scenarios, such as document-level translation and long dialogues, the computational overhead increases quadratically with the sequence length, and the key-value cache in the inference phase also increases accordingly. Complexity has become the main limitation of long context capabilities. Existing optimizations fall into two categories: system-level acceleration improves engineering efficiency but has not broken through the limitations. Upper bound; architectural improvements, lack of global dependency modeling capabilities, expressive loss, or limitations imposed by gradient problems all make it difficult to balance efficiency and performance; state-space models, which have emerged in recent years, achieve linear reasoning complexity and long-range dependency modeling, but still have limitations: pure In global retrieval tasks such as question answering, its performance is inferior to that of the full attention model; a single architecture cannot adapt to the short, medium and long-term multi-scale memory requirements; existing hybrid architectures use static combinations and lack dynamic adaptive capabilities. In summary, existing technologies face challenges in long text processing, including complexity bottlenecks, insufficient multi-scale memory adaptation, and poor mechanism coordination. There is an urgent need for a new architecture that can achieve breakthroughs in both efficiency and performance. To address the aforementioned technical challenges, a natural language processing method based on an adaptive dual-memory hybrid architecture is proposed. Summary of the Invention

[0003] In view of this, the present invention provides a natural language processing method based on an adaptive dual-memory hybrid architecture to solve or alleviate the technical problems existing in the prior art, and at least provides a beneficial option.

[0004] The technical solution of this invention is implemented as follows: A natural language processing method based on an adaptive dual-memory hybrid architecture, comprising the following steps: Embedding transformation: The input text word sequence is obtained and transformed into an embedding representation through the embedding layer, providing input for the adaptive dual-memory hybrid architecture model; Complexity evaluation: Based on the embedded representation, the sequence complexity score is calculated using a sequence complexity estimator with a dynamic memory hierarchical mechanism; Memory allocation: Based on the complexity score and sequence length, the lengths of short-term working memory, medium-term associative memory, and long-term compressed memory are allocated through the dynamic memory stratification mechanism; Sequence segmentation: The embedded representation is segmented into corresponding short-term, medium-term, and long-term memory sequences according to the three-level memory length; Short-term processing: The short-term memory sequence is processed through a hybrid attention mechanism between intra-block and inter-block operations of the short-term working memory module to obtain the attention output; Intermediate processing: The intermediate memory sequence is processed using a selective state-space model of the intermediate associative memory module to obtain... Output and hidden states; Long-term processing: The long-term memory sequence is compressed using a long-term compressed memory module to obtain a compressed memory bank, and then the long-term memory results are retrieved. Bidirectional fusion: Based on the hidden state and attention output, a bidirectional feedback and adaptive fusion is performed through a bidirectional feedback fusion mechanism of attention and state space model to obtain the fused output; Forward optimization: The fused output superimposed with the long-term memory result is passed through the residual connection, layer normalization and feedforward network of the complete adaptive dual-memory hybrid block forward propagation to obtain the final sequence representation; Task output: Input the final sequence representation into the output layer, execute downstream tasks, and output the processing results.

[0005] More preferably, the specific composition of the adaptive dual-memory hybrid architecture model includes: an overall encoder-decoder structure, with core components being a dynamic memory allocator, an adaptive dual-memory hybrid block stack, a neural memory compression module, and an output layer; the adaptive dual-memory hybrid block stack contains N layers of adaptive dual-memory hybrid blocks; the dimension of the embedding representation is defined as... ,in For sequence length, The model hides dimensions; each layer of the adaptive dual-memory hybrid block integrates a short-term working memory module, a medium-term associative memory module, and a bidirectional feedback fusion layer of attention and state space model, and includes residual connections and feedforward network structures.

[0006] More preferably, the sequence complexity estimator of the dynamic memory hierarchical mechanism is specifically implemented including: Local information entropy calculation: dividing the sequence into Each block has a length of [number]. ; Calculate the first The mean vector of each block: ; in For the first Embedding vectors of 1 word element For the first The mean vector of each block; Calculate the normalized attention weights for intra-block lexical terms: ; in as a word element In block Normalized attention weights in the molecule as a word element The similarity score to the block mean, with the normalization factor in the denominator. This is the scaling factor; Calculate the information entropy of the k-th block: ; in For the first The information entropy of each block reflects the uniformity of attention distribution among the lexical units within the block. Calculate the global average entropy: ; in The average information entropy of all blocks represents the average information density of the entire sequence; Long-range dependency strength calculation: Define the cosine similarity function: ; in For vectors and cosine similarity, For vectors The L2 norm and cosine similarity measure the directional similarity between two vectors, with values ​​ranging from 1 to 2. ; Set a long-range distance threshold τ and calculate the long-range dependency strength: ; in Long-range dependency strength measures the average similarity between words located far apart. The larger the value, the stronger the cross-distance dependency between the sequences; Overall complexity score calculation: ; in and For balance coefficient, Used to normalize the entropy value to Interval and sequence complexity scores .

[0007] More preferably, the three-layer memory length allocation of the dynamic memory hierarchy mechanism is specifically implemented as follows: Short-term working memory length allocation: ; in The length of short-term working memory, Indicates rounding up. Indicates rounding down; the short-term working memory length The value range is 256 to 512, and does not exceed the total length of the sequence. 30%; Intermediate associative memory length allocation: ; The length of intermediate-term associated memory The value range is 2K~8K, and satisfies ; Long-term compressed memory length allocation: ; The length of long-term compressed memory The remaining part after deducting the short-term and medium-term memory lengths from the total sequence length; Integrity constraint: The length of the three-layer memory satisfies .

[0008] More preferably, the intra-block and inter-block hybrid attention mechanism of the short-term working memory module is specifically implemented as follows: Short-term memory sequence preprocessing: The short-term memory sequence is ; Divide it into Each block has a length of [number]. , The value ranges from 64 to 128; Intra-block precise attention calculation: for the first block The query matrix, key matrix, and value matrix are obtained through linear projection: ; in Dimensions for each attention head; Compute the attention output within the block: ; in For attention score matrix, The function is row-normalized to obtain the attention weight matrix; this step has a time complexity of O(n log n). ; Inter-block kernelized attention computation, defining the feature mapping function: ; in To obtain from the standard Gaussian distribution Sampling orthogonal random vectors, The value is 256. ; piece For all other blocks Attention approximation calculation: ; Reduce cross-block attention complexity from Down to ; Hybrid attention output, combining intra-block and inter-block attention output: ; in These are learnable hybrid weights; Multi-head hybrid attention extension, using Each attention head performs parallel computation, and each attention head is processed independently; the outputs of all attention heads are concatenated and a linear projection is performed. ; in To output the projection matrix, we finally obtain the attention output. .

[0009] More preferably, the selective state-space model of the intermediate-term associative memory module is specifically implemented including: Basic definition of state-space model: ; in It is in a hidden state. The value is 16; The state transition matrix is ​​initialized using HiPPO to obtain long-range memory capability; For the input projection matrix; To output the projection matrix; Selective parameter generation dynamically generates the input and output projection matrices based on the current input lexical units. ; in , , For linear projection layers; Generate time step parameters: ; in ; Discretization is performed, and the discretized state transition matrix and input projection matrix are calculated using the zero-order preservation method: ; in , Matrix index Efficient computation using diagonalization or Padé approximation methods. It is the identity matrix; Selective state update and output: ; Output calculation formula Parallel computing optimization, among which During the training phase, the intermediate memory sequence Through the parallel prefix sum algorithm Achieving parallel computing: ; in This represents element-wise multiplication, with a training complexity of O(n). The reasoning phase employs sequential recursive calculations, with each step having a complexity of O(n log n). Only a hidden state of constant size needs to be maintained. Memory usage is independent of sequence length; Multi-head state-space model extension adopts Each independent state-space model head is computed in parallel. The value is 8, each Heads are processed independently; splice all The head outputs and performs a linear projection: ; in To output the projection matrix, This is the hidden state of the previous layer, and ultimately we get... Output and the updated hidden state .

[0010] More preferably, the specific implementation of the long-term compressed memory module includes: The neural memory compression algorithm takes a long-term memory sequence as input. The goal is to encode a fixed-size memory. ,in This represents the number of memory slots, with a value of 512. To compress the dimension, the value is 256; Semantic clustering, for Perform K-means clustering algorithm, based on cosine similarity of word vectors Each word element is divided into Given clusters, obtain the cluster index set. ,in For the first Each cluster contains a lexical index; Attention pooling, for each cluster Calculate the attention weights of lexical units within the cluster: ; in The pooling weight vector is a learnable vector, and the denominator ensures that all weights within the cluster sum to 1. Generate cluster representative vectors: ; in Attention weighting is used to highlight the semantic information of important terms within a cluster; autoencoder compression is employed, using a two-layer MLP as the encoder to compress the cluster representative vector. ; in The encoder parameters ultimately yield the compressed memory. ; The memory retrieval mechanism uses a two-layer MLP as a decoder to map the compressed memory back to the original dimension: ; in ; Query vector generation, using short-term memory sequences The mean is used as the query vector: ; Similarity calculation: Calculate the cosine similarity between the query vector and each decoded memory. ; in For the first The similarity between the memory and the query vector; Weighted retrieval involves performing Softmax normalization on the similarity scores and then summing them using weighted methods. ; in The similarity vector is used to obtain the long-term memory result. .

[0011] More preferably, the specific implementation of the bidirectional feedback fusion mechanism of attention and state space model includes: Forward feedback: Projecting the hidden state output by the intermediate associative memory module. Projected onto the key-value space of attention: ; in For the feedback projection matrix, ; Key-value augmentation concatenates the original key-values ​​of the attention layer with the projected feedback key-values ​​along the sequence length dimension: ; in For the original attention key value, To enhance the key-value pair; Enhanced attention computing: ; in The attention query matrix ultimately yields the enhanced attention output. ; Reverse feedback: Gated vector generation, outputting attention. Gated vectors are generated using linear projection and Sigmoid activation: ; in This is the Sigmoid function, with an output range of (0, 1). For linear projection layers, the dimension is mapped from d to... ; For a moment The gate vector; Input modulation, combining the gate vector with After element-wise multiplication of the previous hidden state, the current input words are added: ; in For modulated input, This is the hidden state from the previous moment. This represents element-wise multiplication; Modulation The output will be Input a selective state-space model to obtain the modulated... Output ; Adaptive fusion weight learning: Feature concatenation enhances the attention output and modulated post-processing. The output is concatenated along the feature dimension: ; in ; Meta-network computation learns fusion weights through a two-layer MLP: ; in For dimensionality reduction layer, For weighted prediction layer, The activation function for the Gaussian error linear unit; To normalize the fusion weights, satisfy ; Weighted fusion: ; The final fused output is obtained .

[0012] More preferably, the forward propagation process of the complete adaptive dual-memory hybrid block specifically includes: Layer normalization: representation of the input sequence Execution layer normalization: ; in Normalize the feature dimensions of each word to avoid gradient vanishing; Dynamic sequence segmentation: allocated according to the memory allocation step. ,Will Divided into The segmentation function is ; Multi-module processing integration: Short-term, medium-term, and long-term memory processing are performed separately to obtain... ; Bidirectional feedback fusion: Performing forward feedback, backward feedback, and adaptive fusion to obtain... ; Residual Connections and Long-Term Memory Superposition: The fused output, long-term memory result, and original input are superimposed through residual connections. ; in It is superimposed onto each word element position through a broadcast mechanism; Secondary level normalization: for Execution layer normalization: ; Feedforward network processing: A two-layer MLP is used as the feedforward network. ; in ; Final output: Superimposed feedforward network output and : ; The final sequence representation is obtained. .

[0013] More preferably, the specific implementation of the complexity and efficiency optimization includes: Training complexity optimization: The short-term working memory module has a complexity of [missing information]. The training complexity of the intermediate-term associative memory module is The long-term compressed memory module has a complexity of 1. The overall training complexity is Quasi-linear order of magnitude; Reasoning complexity optimization: The short-term working memory module has a complexity of O(n). The inference complexity of the intermediate-term associative memory module is At each step, the retrieval complexity of the long-term compressed memory module is... The overall reasoning complexity is Linear order of magnitude; Training phase: The AdamW optimizer is used, with cross-entropy loss. ; Backpropagation updates parameters, where ; Inference phase: Initialize and generate a word list Iterative prediction of the next word: ; Update input sequence until generated Mark the end, output .

[0014] The embodiments of the present invention have the following advantages due to the adoption of the above technical solutions: This invention maintains accurate modeling in the critical short-term window and employs an efficient mechanism in the medium- to long-term window, achieving an optimal trade-off between efficiency and performance. It utilizes a three-layer memory system: capturing active information in the short term, maintaining semantic relationships in the medium term, and compressing historical knowledge in the long term. This achieves organic unity of multi-scale memory, bidirectional feedback, and adaptive fusion, enabling different mechanisms to complement and dynamically collaborate, automatically reaching optimal configuration in different task scenarios. This overcomes the quadratic complexity bottleneck of traditional Transformers, reducing training complexity to a minimum. The reasoning complexity is Training and inference speeds are improved, inference memory usage is reduced, and it can efficiently handle extremely long texts with more than 32K words; through attention and... It features bidirectional feedback fusion, balancing global retrieval and long-range modeling capabilities; it improves performance for long text tasks, while maintaining short text performance within ±2% of advanced Transformers, achieving a balance between long and short text performance; it dynamically allocates short, medium, and long-term memory resources based on sequence complexity assessment, saving computation for simple texts and enhancing capabilities for complex texts, avoiding resource waste, and adapting to various scenarios such as dialogue and code generation; it supports complex scenarios such as document-level translation, long dialogues, and large-scale code generation, breaking through the context length limitations of traditional models and providing technical support for the deep implementation of natural language processing.

[0015] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the invention will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the natural language processing method based on an adaptive dual-memory hybrid architecture according to the present invention. Figure 2 This is a flowchart illustrating the complexity evaluation process of the natural language processing method based on an adaptive dual-memory hybrid architecture, as described in this invention. Figure 3 This is a flowchart of the memory allocation sub-method of the natural language processing method based on the adaptive dual-memory hybrid architecture of the present invention. Detailed Implementation

[0018] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0019] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0020] like Figure 1-3 As shown, this embodiment of the invention provides a natural language processing method based on an adaptive dual-memory hybrid architecture, including the following steps: Embedding transformation: The input text word sequence is obtained and transformed into an embedding representation through the embedding layer, providing input for the adaptive dual-memory hybrid architecture model; The adaptive dual-memory hybrid architecture model adopts an encoder-decoder structure, and its core consists of a dynamic memory allocator, an adaptive dual-memory hybrid block stack, a neural memory compression module, and an output layer. Let the input sequence be ,in: Indicates the sequence length; This indicates the hidden dimension of the model; The model is stacked in total Layer-adaptive dual-memory hybrid block, final output sequence representation Used for downstream tasks; Each adaptive dual-memory hybrid block contains three core sub-modules: Short-term working memory module: Processes the most recently active lexical units based on a hybrid intra-block and inter-block attention mechanism; Medium-term associative memory module: Processes semantic associations of medium span based on a selective state-space model; Bidirectional Feedback Fusion Layer: Enables bidirectional information flow and adaptive weighting between attention and state space models.

[0021] Complexity evaluation: Based on the embedding representation, a sequence complexity estimator with a dynamic memory hierarchical mechanism is used to calculate the sequence complexity score; The sequence complexity estimator performs dynamic analysis on the input sequence; Embedded representation of input sequence Calculate local and global statistics; Local information entropy calculation: Split the sequence into Each block has a length of [number]. , for the For each block, first calculate the mean vector of that block: ; in: For the first The mean vector of each block; The summation range covers all tokens within the block; Then, normalized weights are obtained through temporary calculation using self-attention. : ; in: as a word element In block Normalized attention weights in the data; molecular as a word element Similarity score to the block mean; The denominator is a normalization factor to ensure that the sum of all weights is 1; This is a scaling factor to prevent the gradient from vanishing due to an excessively large dot product; Calculate the first Information entropy of each block : ; in: For the first Information entropy of each block; Information entropy reflects the uniformity of attention distribution among lexical units within a block; A larger value indicates a more uniform distribution of information and higher complexity. The global average entropy is defined as: ; in The average information entropy of all blocks represents the average information density of the entire sequence; Long-range dependency strength calculation: To calculate the similarity distribution of word pairs at different distances in a sequence, define the cosine similarity function: ; in: For vectors and Cosine similarity; For vectors The L2 norm; Cosine similarity measures the directional similarity between two vectors, and its value ranges from 1 to 2. ; Long-range dependency strength Distance Average similarity of word pairs at different positions: ; in: This is the long-range distance threshold; Measure the average similarity between words that are located far apart; Long-range dependency strength The larger the value, the stronger the cross-distance dependency between the sequences; Overall complexity score: A sequence complexity score is defined by combining local entropy and long-range dependency strength. : ; in: and For balance coefficient, , ; Used to normalize the entropy value to interval; Sequence complexity score ; A larger value indicates higher sequence complexity and requires more computational resources.

[0022] Memory allocation: Based on complexity score and sequence length, the lengths of short-term working memory, medium-term associative memory, and long-term compressed memory are allocated through a dynamic memory stratification mechanism; Score based on sequence complexity and sequence length Dynamically allocate the processing range of the three-layer memory; Short-term working memory allocation: Short-term working memory is used to process recently active terms, capture local dependencies and short-distance interactions, and allocate length. The calculation formula is: ; in: The length of short-term working memory; Indicates rounding up; Indicates rounding down; This formula ensures that the short-term memory length is between 256 and 512, and does not exceed 30% of the total sequence length; When sequence complexity is high, short-term memory capacity increases to process more active information; When the sequence is simple, short-term memory is kept at a minimum configuration to save computation. Intermediate associative memory allocation: Medium-range associative memory is used to process semantic associations and thematic coherence across a medium span, allocating length... The calculation formula is: ; in: The length of intermediate associative memory; This formula ensures that the medium-term memory length is between 2K and 8K. constraint Ensure that medium-term memory does not overlap with short-term memory; Complexity scores have a greater impact on intermediate memory because complex texts typically require stronger intermediate dependency modeling capabilities. Long-term compressed memory allocation: Long-term compressed memory is used to process historical content that exceeds the capacity of medium-term memory, and its length is allocated accordingly. For the remaining part: ; in: To compress the length of long-term memory; The allocation of the three-layer memory satisfies the integrity constraint: .

[0023] Sequence segmentation: The embedding representation is segmented into corresponding short-term, medium-term, and long-term memory sequences according to the three-level memory length; Short-term processing: The short-term memory sequence is processed through a hybrid attention mechanism between intra-block and inter-block operations of the short-term working memory module to obtain the attention output; Intra-block and inter-block hybrid attention mechanism: Suppose a short-term memory sequence Divide it into Each block has a length of [number]. ; Precise attention within blocks: For the block The query matrix is ​​obtained through linear projection. Key matrix Value matrix : ; in: These are the query, key, and value projection matrices, respectively. The dimension representing each attention head; ; Intra-block attention output Calculated using standard self-attention: ; in: For block-level attention output; This is the attention score matrix; Normalize by row to obtain the attention weight matrix; This operation captures fine-grained interactions between lexical units within a block, with a complexity of O(n log n). ; Inter-block nucleation attention: To efficiently model cross-block information, orthogonal random features are used for kernel approximation, and a feature mapping function is defined. : ; in: To obtain from the standard Gaussian distribution Sampling orthogonal random vectors; For feature dimensions; The result of feature mapping; Approximate kernel function using random Fourier features ; piece For all other blocks The attention is approximately: ; in: For inter-block attention output; Feature mapping for the query; Aggregation of key-value pairs; Through feature mapping , will the original Cross-block attention computation reduced to ; Mixed output: Combining intra-block and inter-block attention, the hybrid output is: ; in: These are learnable hybrid weights; Initialize to With a preference for precise intra-block attention; During training, the model automatically learns the optimal intra-block and inter-block balance; Multi-headed mixed attention: Expand to Each attention point, the final output is: ; in: Each Calculate independently; To output the projection matrix; This indicates concatenation along the feature dimension; The multi-head mechanism enables the model to focus on different semantic subspaces in parallel.

[0024] Intermediate processing: The intermediate memory sequence is processed using a selective state-space model of the intermediate associative memory module to obtain... Output and hidden states; State-space model basics: Traditional recurrent neural networks suffer from the vanishing gradient problem, which limits the effective memory length; although long short-term memory networks introduce gating mechanisms, their serial computation characteristics prevent parallel training; state-space models model discrete time series as a discretization of a continuous state space. The standard state-space model is defined as follows: ; ; in: It is in a hidden state; This is the state transition matrix; For the input projection matrix; To output the projection matrix; State transition matrix Long-range memory capability is acquired through special initialization; Selective mechanism: Traditional state-space model , , Since information remains unchanged throughout the sequence and cannot be dynamically adjusted for selective retention or forgetting based on input content, this invention introduces an input-dependent selectivity parameter: ; ; in: and From the current input Dynamically generated; This enables the model to selectively encode and output information based on the content; The time step parameter controls the state update speed; make sure ; The discretized state transition parameters are calculated using the zeroth-order preservation method: ; in: This is the discretized state transition matrix; To discretize the input projection matrix; Matrix index Efficient computation using diagonalization or Padé approximation methods; It is the identity matrix; The state update of the selective state-space model is as follows: ; ; in: For a moment The hidden state; For a moment The output; parallel computing: Training phase: Training the entire intermediate memory sequence Efficient parallel computation is achieved through a parallel prefix sum algorithm, for each position. Pre-calculated coefficient matrix and Then, the calculation is performed recursively using a binary tree structure: ; in: This represents element-wise multiplication; The algorithm will The sequential recursion is transformed into Deep parallel computing; The training complexity is ; Inference phase: Sequential recursion requires constant time per step, maintaining a hidden state of constant size. Memory usage is independent of sequence length, and the reasoning complexity is O(n log n). Each step; Multi-head state-space model layer: Similar to multi-head attention, using A separate state-space model head: ; in: To output the projection matrix; Each Each head independently models different temporal patterns; This indicates splicing along the feature dimension.

[0025] Long-term processing: The long-term memory sequence is compressed using a long-term compressed memory module to obtain a compressed memory bank, and then the long-term memory results are retrieved. Neural memory compression algorithm: For historical sequences exceeding the capacity of medium-term memory It is encoded into a fixed-size memory using a neural compression algorithm. ,in For the number of memory slots, To compress dimensions; Simple average pooling loses positional information, max pooling only retains extreme value features, truncation directly discards early content, and neural compression extracts the most important semantic representations through learning. Semantic clustering: First of all The K-means clustering algorithm is applied to group words with similar semantics: ; in: Indicates the first Each cluster contains a set of lexical indexes; Clustering is based on the cosine similarity of word vectors; Will Each word element is divided into A cluster; Attention pooling: For each cluster We use learnable attention pooling to generate representative vectors and compute attention weights: ; in: The pooling weight vector is a learnable vector. For clusters Chinese word elements Attention weights; The denominator ensures that the sum of all weights within the cluster is 1; Clusters represent vectors as weighted sums: ; in: For the first The representative vector of each cluster; By using attention-weighted algorithms, important lexical terms receive higher weights. Autoencoder compression: Further compression to a lower-dimensional space via an autoencoder: ; in: It is a two-layer MLP: ; Indicates encoder parameters; This is a compressed memory representation; Compressed memory It retains the core semantic information of the cluster while significantly reducing storage overhead; Memory retrieval mechanism: When processing the current sequence, based on the query vector Retrieve relevant information from the compressed memory; Similarity calculation: First, the compressed representation is mapped back to the original dimension using a decoder: ; in: It is a two-layer MLP: ; This is the decoded memory representation; Calculate the cosine similarity between the query vector and each compressed memory: ; in: query vector With the The similarity of individual memories; It is an L2 norm; Weighted search: The retrieved memories are a weighted sum of all compressed memories: ; in: This is a similarity vector; Ensure weight normalization; The retrieval vector contains historical information most relevant to the current query.

[0026] Bidirectional fusion: Based on the hidden state and attention output, a bidirectional feedback and adaptive fusion mechanism is used to perform bidirectional feedback fusion through the attention and state space model to obtain the fused output; Bidirectional feedback loop design: The bidirectional feedback fusion mechanism enables bidirectional information flow and dynamic collaboration between the attention and state space models; Forward feedback from state-space model to attention: Hidden state sequence output by the state-space model layer A compressed representation containing long-range dependencies is projected onto the key-value space of the attention: ; in: For the feedback projection matrix; The projected key value; This projection transforms the hidden states of the state-space model into a format usable by the attention mechanism; During computation, the attention layer concatenates the original key values ​​and the feedback key values ​​from the state-space model. ; in: This indicates concatenation along the sequence length dimension; These are the original key values ​​of the attention layer; The enhanced key-value pair; The enhanced attention calculation is as follows: ; in: The query matrix for the attention layer; To enhance attention output; Through this forward feedback, the attention mechanism can not only retrieve information from the current sequence, but also access the long-range historical states encoded by the state-space model, thus enhancing its global context awareness. Attention to the backfeedback of the state-space model: Attention layer output The results, encompassing global information retrieval and fine-grained interactions, are used as selective gating signals for the state-space model. ; in: The Sigmoid activation function has an output range of... ; For a moment The gate vector; For linear projection layers, the dimensions are changed from... Mapped to ; What information does the gated vector control state-space model retain or discard? The input to the modulated state-space model is: ; in: The modulated input; The hidden state of the state-space model at the previous moment; This represents element-wise multiplication; Through this reverse feedback, the global perspective of attention guides the selective parameters of the state-space model, enabling it to dynamically adjust the information retention strategy according to the global context and optimize the accuracy of long-range dependency modeling. Adaptive fusion weight learning: To dynamically balance the contributions of attention and the state-space model, a meta-network is designed to learn position-related fusion weights; the input to the meta-network is the concatenation of the outputs of two modules. ; in: This is the output of the state-space model; Features after splicing; This indicates concatenation along the feature dimension; The fusion weights are calculated using a two-layer perceptron MLP: ; in: For normalized fusion weights, satisfy ; For dimensionality reduction; For weighted prediction layer; The activation function for the Gaussian error linear unit; Ensure that the two weights are normalized; The final fusion output is: ; in: The output after fusion; Metanetworks can adaptively adjust the weights of the two mechanisms based on the semantic features of the current position; Increase attention weight in locations where fine-grained global retrieval is required; Increase the weight of the state-space model in locations where efficient long-range modeling is required.

[0027] Forward optimization: The fused output superimposed with the long-term memory result is passed through the residual connection, layer normalization and feedforward network of the complete adaptive dual-memory hybrid block forward propagation to obtain the final sequence representation; The complete forward propagation process for a single adaptive dual-memory hybrid block is as follows: Input: Sequence representation The hidden state of the state space model of the previous layer ; Step 1: Layer Normalization: ; in Normalize the features of each lexical unit; Step 2: Dynamic memory segmentation: According to pre-calculated , , Split sequence: ; in: This refers to the short-term memory portion; This is the intermediate memory section; This is the long-term memory portion; Step 3: Short-term memory processing: Processing short-term memory through mixed attention: ; in For short-term memory attention output; Step 4: Intermediate memory processing: Processing intermediate memory using a selective state-space model: ; in: This is the output of the state-space model; This is the updated hidden state; Step 5: Long-term memory compression and retrieval: Compressing long-term memory: ; in For the compressed memory; Retrieve relevant information using the mean of short-term memory as the query vector: ; in: The average vector of short-term memory; For retrieved long-term memory information; Step 6: Two-way feedback fusion: Attention is obtained from the state-space model feedback: ; Attention gating is achieved through state-space models: ; Adaptive fusion: ; Output ; Step 7: Residual Connections and Feedforward Networks: Incorporate long-term memory retrieval results and residual connections: ; Residual connections maintain information flow, and long-term memory retrieves results. Added to each location via broadcast; Second layer normalization: ; Feedforward network processing: ; in: It is a two-layer MLP: ; Use the GELU activation function; Follow the standard Transformer feedforward network design; Output: Updated sequence representation and hidden states in state-space models .

[0028] Task output: Input the final sequence representation into the output layer, execute downstream tasks, and output the processing results.

[0029] In one embodiment, the complete algorithm flow is as follows: Training phase algorithm: enter: training set It contains several input / output pairs; Hyperparameters: Model Dimensions Number of floors Number of attention heads State dimension Learning rate Batch size ; Algorithm flow: Initialize model parameters; Randomly initialize all weight matrices wait; State transition matrix of state-space model Initialize using HiPPO; The optimizer is set to AdamW. Weight decay = 0.01; Train a loop for each epoch; For each batch in the training set : a. Calculate sequence complexity: Calculate ; b. Dynamic memory allocation: calculation , , ; c. Initialize the hidden state: ; d. Layer-by-layer forward propagation: Execute adaptive dual-memory hybrid block layer; enter: ; Output: ; e. Output layer computation: ; f. Loss calculation: ; g. Backpropagation and parameter update: Calculate the gradient: ; AdamW Update: ; Save the model: Save the trained parameters ; Output: The parameters of the trained model; Inference phase algorithm: Input: Sequence to be processed ; Algorithm flow: Load model parameters: Load the trained model parameters ; Sequence preprocessing; initialization: ; Computational complexity: ; Dynamic allocation: ; Autoregressive generation; initialization: ; right arrive : a. Layer-by-layer forward propagation: Execute adaptive dual-memory hybrid block layer; b. Predict the next word: ; ; c. Update the input sequence: ; ; d. Termination condition: if for Then break; Return result: Return ; Output: The generated word sequence or classification result.

[0030] In one embodiment, time complexity analysis is performed: Short-term working memory module: Intra-block attention: Complexity per block , The total complexity of each block is... ; because and It is a constant, and the actual complexity is O(n). ; Inter-block nucleation attention: ,in It is a constant; Overall short-term memory complexity: ; Intermediate-term associative memory module: Training phase: Parallel scanning algorithm complexity ; Reasoning phase: Sequential recursion at each step Maintain a fixed-size hidden state; Long-term compressed memory module: Clustering and compression: ,in It is a constant; Search: , It is a constant; Overall complexity: Training complexity: ; because and The linear coefficients are reduced by the compression operation, and the dominant term is... That is, quasi-linear complexity; Reasoning complexity: Linear complexity.

[0031] In one embodiment, space complexity analysis is performed: Short-term working memory module: Key-value pairs that need to store attention within the block: ; Because kernelized approximation is used, there is no need to store the complete cross-block attention matrix; Intermediate-term associative memory module: Only the hidden state needs to be maintained: It is independent of the sequence length; Long-term compressed memory module: Compressed memory: Fixed size; Overall space complexity: ; because , and It can reach over 32K, with a significantly lower space footprint than traditional Transformers. or key-value cache .

[0032] In operation, this invention: acquires the input text word sequence, converts it into a fixed-dimensional embedding representation through an embedding layer, providing basic data for subsequent processing; based on the embedding representation, calculates local information entropy and long-range dependency strength, and fuses them to obtain a complexity score; according to the complexity score and sequence length, allocates short-term, medium-term, and long-term memory lengths, and segments the embedding representation into corresponding memory sequences accordingly; the short-term sequence is processed using a hybrid attention model of intra-block precision and inter-block kernelization to obtain the attention output; the medium-term sequence is processed using a selective state-space model to obtain... Output and hidden state; long-term sequences are clustered, attention pooled, compressed by an autoencoder, and then retrieved using the short-term mean to obtain the long-term memory result; Latent state feedback enhances attention key value, and attention output gating modulation The input is the weights of the meta-network, which are then fused to produce the output. The fused result is superimposed on long-term memory, passed through residual connections, layer normalization, and a feedforward network to obtain the final sequence representation. The final representation is then input into the output layer to perform downstream tasks such as translation and generation, and the processing result is output.

[0033] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in the present invention, and these should all be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A natural language processing method based on an adaptive dual-memory hybrid architecture, characterized in that, Includes the following steps: Embedding transformation: The input text word sequence is obtained and transformed into an embedding representation through the embedding layer, providing input for the adaptive dual-memory hybrid architecture model; Complexity evaluation: Based on the embedded representation, the sequence complexity score is calculated using a sequence complexity estimator with a dynamic memory hierarchical mechanism; Memory allocation: Based on the complexity score and sequence length, the lengths of short-term working memory, medium-term associative memory, and long-term compressed memory are allocated through the dynamic memory stratification mechanism; Sequence segmentation: The embedded representation is segmented into corresponding short-term, medium-term, and long-term memory sequences according to the three-level memory length; Short-term processing: The short-term memory sequence is processed through a hybrid attention mechanism between intra-block and inter-block operations of the short-term working memory module to obtain the attention output; Intermediate processing: The intermediate memory sequence is processed using a selective state-space model of the intermediate associative memory module to obtain... Output and hidden states; Long-term processing: The long-term memory sequence is compressed using a long-term compressed memory module to obtain a compressed memory bank, and then the long-term memory results are retrieved. Bidirectional fusion: Based on the hidden state and attention output, a bidirectional feedback and adaptive fusion is performed through a bidirectional feedback fusion mechanism of attention and state space model to obtain the fused output; Forward optimization: The fused output superimposed with the long-term memory result is passed through the residual connection, layer normalization and feedforward network of the complete adaptive dual-memory hybrid block forward propagation to obtain the final sequence representation; Task output: Input the final sequence representation into the output layer, execute downstream tasks, and output the processing results.

2. The natural language processing method based on an adaptive dual-memory hybrid architecture according to claim 1, characterized in that: The specific structure of the adaptive dual-memory hybrid architecture model includes: an overall encoder-decoder structure, with core components being a dynamic memory allocator, an adaptive dual-memory hybrid block stack, a neural memory compression module, and an output layer; the adaptive dual-memory hybrid block stack contains N layers of adaptive dual-memory hybrid blocks; the dimension of the embedding representation is defined as... ,in For sequence length, The model hides dimensions; each layer of the adaptive dual-memory hybrid block integrates a short-term working memory module, a medium-term associative memory module, and a bidirectional feedback fusion layer of attention and state space model, and includes residual connections and feedforward network structures.

3. The natural language processing method based on an adaptive dual-memory hybrid architecture according to claim 1, characterized in that: The specific implementation of the sequence complexity estimator for the dynamic memory hierarchical mechanism includes: Local information entropy calculation: dividing the sequence into Each block has a length of [number]. ; Calculate the first The mean vector of each block: in For the first Embedding vectors of 1 word element For the first The mean vector of each block; Calculate the normalized attention weights for intra-block lexical terms: in as a word element In block Normalized attention weights in the molecule as a word element The similarity score to the block mean, with the normalization factor in the denominator. This is the scaling factor; Calculate the information entropy of the k-th block: in For the first The information entropy of each block reflects the uniformity of attention distribution among the lexical units within the block. Calculate the global average entropy: in The average information entropy of all blocks represents the average information density of the entire sequence; Long-range dependency strength calculation: Define the cosine similarity function: in For vectors and cosine similarity, For vectors The L2 norm and cosine similarity measure the directional similarity between two vectors, with values ​​ranging from 1 to 2. ; Set a long-range distance threshold τ and calculate the long-range dependency strength: in Long-range dependency strength measures the average similarity between words located far apart. The larger the value, the stronger the cross-distance dependency between the sequences; Overall complexity score calculation: in and For balance coefficient, Used to normalize the entropy value to Interval and sequence complexity scores .

4. The natural language processing method based on an adaptive dual-memory hybrid architecture according to claim 1, characterized in that: The specific implementation of the three-layer memory length allocation of the dynamic memory hierarchy mechanism includes: Short-term working memory length allocation: in The length of short-term working memory, Indicates rounding up. Indicates rounding down; the short-term working memory length The value range is 256 to 512, and does not exceed the total length of the sequence. 30%; Intermediate associative memory length allocation: The length of intermediate-term associated memory The value range is 2K~8K, and satisfies ; Long-term compressed memory length allocation: The length of long-term compressed memory The remaining part after deducting the short-term and medium-term memory lengths from the total sequence length; Integrity constraint: The length of the three-layer memory satisfies .

5. The natural language processing method based on an adaptive dual-memory hybrid architecture according to claim 1, characterized in that: The specific implementation of the intra-block and inter-block hybrid attention mechanism of the short-term working memory module includes: Short-term memory sequence preprocessing: The short-term memory sequence is Divide it into Each block has a length of [number]. , The value ranges from 64 to 128; Intra-block precise attention calculation: for the first block The query matrix, key matrix, and value matrix are obtained through linear projection: in Dimensions for each attention head; Compute the attention output within the block: in For attention score matrix, The function is row-normalized to obtain the attention weight matrix; this step has a time complexity of O(n log n). ; Inter-block kernelized attention computation, defining the feature mapping function: in To obtain from the standard Gaussian distribution Sampling orthogonal random vectors, The value is 256. ; piece For all other blocks Attention approximation calculation: Reduce cross-block attention complexity from Down to ; Hybrid attention output, combining intra-block and inter-block attention output: in These are learnable hybrid weights; Multi-head hybrid attention extension, using Each attention head performs parallel computation, and each attention head is processed independently; the outputs of all attention heads are concatenated and a linear projection is performed. in To output the projection matrix, we finally obtain the attention output. .

6. The natural language processing method based on an adaptive dual-memory hybrid architecture according to claim 1, characterized in that: The selective state-space model of the intermediate-term associative memory module is specifically implemented as follows: Basic definition of state-space model: in It is in a hidden state. The value is 16; The state transition matrix is ​​initialized using HiPPO to obtain long-range memory capability; For the input projection matrix; To output the projection matrix; Selective parameter generation dynamically generates the input and output projection matrices based on the current input lexical units. in , , For linear projection layers; Generate time step parameters: in ; Discretization is performed, and the discretized state transition matrix and input projection matrix are calculated using the zero-order preservation method: in , Matrix index Efficient computation using diagonalization or Padé approximation methods. It is the identity matrix; Selective state update and output: Output calculation formula Parallel computing optimization, among which During the training phase, the intermediate memory sequence Through the parallel prefix sum algorithm Achieving parallel computing: in This represents element-wise multiplication, with a training complexity of O(n). The reasoning phase employs sequential recursive calculations, with each step having a complexity of O(n log n). Only a hidden state of constant size needs to be maintained. Memory usage is independent of sequence length; Multi-head state-space model extension adopts Each independent state-space model head is computed in parallel. The value is 8, each Heads are processed independently; splice all The head outputs and performs a linear projection: in To output the projection matrix, This is the hidden state of the previous layer, and ultimately we get... Output and the updated hidden state .

7. The natural language processing method based on an adaptive dual-memory hybrid architecture according to claim 1, characterized in that: The specific implementation of the long-term compressed memory module includes: The neural memory compression algorithm takes a long-term memory sequence as input. The goal is to encode a fixed-size memory. ,in This represents the number of memory slots, with a value of 512. To compress the dimension, the value is 256; Semantic clustering, for Perform K-means clustering algorithm, based on cosine similarity of word vectors Each word element is divided into Given clusters, obtain the cluster index set. ,in For the first Each cluster contains a lexical index; Attention pooling, for each cluster Calculate the attention weights of lexical units within the cluster: in The pooling weight vector is a learnable vector, and the denominator ensures that all weights within the cluster sum to 1. Generate cluster representative vectors: in Attention weighting is used to highlight the semantic information of important terms within a cluster; autoencoder compression is employed, using a two-layer MLP as the encoder to compress the cluster representative vector. in The encoder parameters are used to ultimately obtain the compressed memory. ; The memory retrieval mechanism uses a two-layer MLP as a decoder to map the compressed memory back to the original dimension: in ; Query vector generation, using short-term memory sequences The mean is used as the query vector: ; Similarity calculation: Calculate the cosine similarity between the query vector and each decoded memory. in For the first The similarity between the memory and the query vector; Weighted retrieval involves performing Softmax normalization on the similarity scores and then summing them using weighted methods. in The similarity vector is used to obtain the long-term memory result. .

8. The natural language processing method based on an adaptive dual-memory hybrid architecture according to claim 1, characterized in that: The specific implementation of the bidirectional feedback fusion mechanism of attention and state space model includes: Forward feedback: Projecting the hidden state output by the intermediate associative memory module. Projected onto the key-value space of attention: in For the feedback projection matrix, Key-value augmentation concatenates the original key-values ​​of the attention layer with the projected feedback key-values ​​along the sequence length dimension: in For the original attention key value, To enhance the key-value pair; Enhanced attention computing: in The attention query matrix ultimately yields the enhanced attention output. ; Reverse feedback: Gated vector generation, outputting attention. Gated vectors are generated using linear projection and Sigmoid activation: in This is the Sigmoid function, with an output range of (0, 1). For linear projection layers, the dimension is mapped from d to... ; For a moment The gate vector; Input modulation, combining the gate vector with After element-wise multiplication of the previous hidden state, the current input words are added: in For modulated input, This is the hidden state from the previous moment. This represents element-wise multiplication; Modulation The output will be Input a selective state-space model to obtain the modulated... Output ; Adaptive fusion weight learning: Feature concatenation enhances the attention output and modulated post-processing. The output is concatenated along the feature dimension: in ; Meta-network computation learns fusion weights through a two-layer MLP: in For dimensionality reduction layer, For weighted prediction layer, The activation function for the Gaussian error linear unit; To normalize the fusion weights, satisfy ; Weighted fusion: The final fused output is obtained .

9. A natural language processing method based on an adaptive dual-memory hybrid architecture according to claim 1, characterized in that: The specific implementation of the forward propagation process of the complete adaptive dual-memory hybrid block includes: Layer normalization: representation of the input sequence Execution layer normalization: in Normalize the feature dimensions of each word to avoid gradient vanishing; Dynamic sequence segmentation: allocated according to the memory allocation step. ,Will Divided into The segmentation function is ; Multi-module processing integration: Short-term, medium-term, and long-term memory processing are performed separately to obtain... ; Bidirectional feedback fusion: Performing forward feedback, backward feedback, and adaptive fusion to obtain... ; Residual Connections and Long-Term Memory Superposition: The fused output, long-term memory result, and original input are superimposed through residual connections. in It is superimposed onto each word element position through a broadcast mechanism; Secondary level normalization: for Execution layer normalization: Feedforward network processing: A two-layer MLP is used as the feedforward network. in ; Final output: Superimposed feedforward network output and : The final sequence representation is obtained. .

10. A natural language processing method based on an adaptive dual-memory hybrid architecture according to claim 1, characterized in that: The specific implementation of the complexity and efficiency optimization includes: Training complexity optimization: The short-term working memory module has a complexity of [missing information]. The training complexity of the intermediate-term associative memory module is The long-term compressed memory module has a complexity of 1. The overall training complexity is Quasi-linear order of magnitude; Reasoning complexity optimization: The short-term working memory module has a complexity of O(n). The inference complexity of the intermediate-term associative memory module is At each step, the retrieval complexity of the long-term compressed memory module is... The overall reasoning complexity is Linear order of magnitude; Training phase: The AdamW optimizer is used, with cross-entropy loss. Backpropagation updates parameters, where ; Inference phase: Initialize and generate a word list Iterative prediction of the next word: Update input sequence until generated Mark the end, output .

Images