An attention-based reading-out and competitive updating-based language model external memory method

Abstract

The application discloses a language model external memory method based on attention reading and competitive updating, and the method comprises the following steps: S1, reading out a memory reading amount from a memory matrix by cross-layer attention reading with an input hidden state of a specified layer as a query, and injecting the layer residual flow by deep routing after residual iterative refinement; S2, after a conversation ends and before the matrix is read out, compressing a token representation sequence of the current conversation into a candidate matrix of the same shape by using a learnable rewriting query; and S3, after the old memory matrix and the candidate matrix are spliced along the slot position, performing a softmax competitive attention update by using a learnable judgment query, so that the new and old contents compete in a zero-sum manner, and the output memory matrix is used as the initial memory for the next conversation. According to the application, cross-conversation long-range information can be accessed with constant overhead without modifying model parameters or external retrieval.

Description

Technical Field

[0001] This invention belongs to the field of artificial neural networks and natural language processing technology, and in particular relates to an external memory method for language models based on attention reading and competitive updating. Background Technology

[0002] In recent years, large-scale pre-trained language models (LLMs) deployed in dialogue scenarios have encountered significant difficulties in "long-range memory" tasks such as long-term conversations, cross-conversation recall, and multi-day accumulated context. Currently, the main existing technical solutions in this field fall into the following three categories: 1. Extended Context Window Scheme: This scheme concatenates historical dialogues as raw text into the input context of the current inference. Assuming the context length is N, the computational complexity of standard self-attention is... This also exhibits superlinear growth in linear attention variants. Furthermore, each inference iteration requires maintaining a key-value cache (KV-cache) in GPU memory, the size of which is proportional to N. Representative existing techniques include: long-context Transformer classes (such as Longformer, BigBird, StreamingLLM, etc.); and methods that extend the window length through position interpolation / extrapolation (such as Position Interpolation, YaRN, etc.). 2. Retrieval-Augmented Generation (RAG) scheme: This scheme stores historical dialogues in an external vector database. During inference, it first vectorizes the current query using an embedding model, then retrieves several historical text fragments using similarity retrieval, and reassembles these fragments into the context window of the language model. Representative existing technologies include traditional RAG pipelines, various retrieval-based dialogue systems, and external components such as "long-term memory plugins." 3. Per-user Fine-tuning scheme: This scheme incrementally fine-tunes the language model ontology weights based on each user's dialogue history, such as full-parameter fine-tuning, LoRA, Adapter and other efficient parameter fine-tuning methods.

[0003] Disadvantages of existing technology: 1. Disadvantages of the extended context window scheme: (1) As the session length increases, the computational power and memory cost of a single inference increases monotonically, exhibiting a quadratic or superlinear pattern, which is economically unsustainable in the long term; (2) The attention mechanism suffers from "information loss in the middle" in the middle of a long context, meaning that the model's utilization of information in the middle of the context decreases significantly; (3) Most pre-trained models have weak extrapolation capabilities for contexts exceeding the window length during training, and forced expansion will introduce distribution shift; (4) Historical information repeatedly enters the context in the form of the original token, lacking an active discrimination mechanism of "what should be remembered and what should be forgotten".

[0004] The disadvantages of the retrieval enhancement generation scheme are: (1) It depends on external independent infrastructure (embedded model, vector library, retrieval pipeline, reordering model), resulting in high system complexity and high deployment and maintenance costs; (2) The retrieval machine and language model are decoupled during the training phase, making it impossible to jointly learn "what content is worth remembering", and the memory strategy can only be designed by manual heuristics; (3) The retrieval results are fed back to the context in the form of natural language text, which still consumes the same token-level computing power as the extended window scheme; (4) The retrieval status of each user needs to be maintained independently outside the language model, which makes it difficult to meet the needs of on-device private memory.

[0005] The disadvantages of the user-level fine-tuning scheme are: (1) The computational cost of updating parameters for each user is huge and it is difficult to scale to a large user group; (2) Fine-tuning is prone to catastrophic forgetting, which damages the general ability learned in the pre-training stage; (3) The size of each user's individual parameter copy is usually much larger than the storage space required for a dialogue history, making it uneconomical to deploy on the edge; (4) Once fine-tuning is completed, the model "solidifies" the user information, making it difficult to actively forget or update a specific memory in fine granularity. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art. This invention enables the language model to access long-range information across sessions with constant computational overhead that is independent of the historical length during forward inference, without modifying the ontology parameters of the pre-trained language model, without relying on any external retrieval system, and without expanding the model context window. The method itself determines which historical information should be retained and which should be overwritten.

[0007] To achieve the above-mentioned objectives, the technical solution provided by this invention patent is as follows: An external memory method for language models based on attention-based reading and competitive updating, the method specifically includes the following steps: S1, Memory-Attention Injection: In each inference session, for one or more specified layers of the language model, the input hidden state of that layer is used as a query. A memory readout is read from the fixed-capacity memory matrix in a cross-layer attention manner. The memory readout is refined at least once using the memory matrix as the key. The refined memory readout is then injected into the residual stream of that layer through a deep routing mechanism. S2, Automatic rewriting: After a session ends and before the memory matrix is ​​read by a subsequent session, a set of learnable rewriting queries is applied to the token representation sequence of the just-ended session, and the information contained in the session is compressed into a candidate memory matrix with the same shape as the memory matrix through cross-layer attention. S3, a two-stage competition, concatenates the old memory matrix and the candidate matrix along the slot dimension into a concatenated matrix of length 2K. A set of learnable decision queries is applied to the concatenated matrix, and competitive attention updates based on softmax normalization are performed. The old slot content and the new candidate content form a zero-sum competition through softmax normalization. The competition result determines the content of each slot in the updated memory matrix, and the updated memory matrix is ​​produced as the initial memory for the next session.

[0008] Furthermore, the language model has L decoder layers, and one or more sets of indices for the injected layers are set. The memory-attention injection in S1 specifically refers to: S11, Configure a learnable projection matrix unique to each injected layer ℓ. , , Let the input hidden state sequence of this layer at the current inference position be... Where N is the length of the current sequence; S12, obtain the memory readout amount of the current layer and current position from the fixed-capacity memory matrix, specifically as follows: ;in For the current input hidden state sequence of this layer, Softmax normalizes it along the K-dimensional axis; S13, will be obtained As the base source for this layer, it is added to the optional source pool of the residual flow. Let the routable source pool accumulated by this layer along the depth direction be: , in This indicates the memory source injected by this method, and b_0 represents the output of the embedding layer. This represents intermediate residual sources from previous decoded blocks; using learnable pseudo-queries. right Softmax weighting is applied to each source in the dataset: , ; and with As the actual input to subsequent sub-layers of this layer.

[0009] Furthermore, S1 also includes: introducing a refinement depth R≥1, and iteratively performing the following steps on r=1,2,...,R: , among which It has independent learnable parameters.

[0010] Furthermore, S2 specifically includes: S21, Configure a set of learnable rewrite query matrices The number of rows is consistent with the number of slots K in M_c; each row of M_in is initialized orthogonally, and the slot position encoding is superimposed on each row during forward propagation to break the permutation symmetry; S22, compressing this session into a candidate memory matrix specifically is as follows: , in This is the token representation sequence for this session. , , For learning projection, This is the result of compression; S23, output E as M_new.

[0011] Furthermore, S2 also includes: introducing an extraction depth L_E≥0, for Iterative execution: , , , Among them each It has independent learnable parameters.

[0012] Furthermore, S3 specifically includes: S31, splice the old memory and the new candidate along the slot axis: ; S32, Configure a set of learnable decision query matrices Its row number and the final output The number of slots K is consistent; M_judge is initialized orthogonally and the slot position code is superimposed; S33, with M_judge as the initial slot state For i=1,...,T, perform the following competitive update, where T is the preset number of iterations: ; ; ; ; ,in , , For learnable projection; Softmax_K represents softmax normalization along the K memory slots, so that the attention quality of each of the 2K input slots (K old slots and K new candidate slots) in the concatenation matrix P is zero-sum distributed among the K memory slots; norm_slot represents weighted average normalization of each memory slot row so that the row sum is 1; GRU is a gated recurrent unit with weights shared across slots and hidden states independent per slot, realizing slot-by-slot retain / write gating.

[0013] Furthermore, S3 also includes: before performing Softmax normalization, for each input column, only the k highest-scoring columns are retained in the K output slots, and the logits of the remaining positions are set to zero. Then perform Softmax; where k is a preset small integer; After performing Softmax normalization, a learnable slot-by-slot gate vector is appended. Gating fusion is performed using the following formula: , where M_judged represents the output after the S33 iteration ends.

[0014] Furthermore, the set of injected layer indices can be any one of the following: injecting all layers, injecting only a few specified layers, or dynamically determining the injected layers according to a learnable layer selection gate.

[0015] Furthermore, the learnable parameters introduced by this method are obtained through end-to-end language modeling loss training under the condition that the language model ontology parameters are frozen; and the total training loss also includes an auxiliary loss with a lower bound on the proportion of deep routing memory.

[0016] in For the set of all injected layers, For the first Layer depth routing assigns softmax weights to memory source b_{-1} The average value across all token positions in the current training batch, τ∈(0,1) is the preset lower threshold, ReLU(x)=max(x,0); the total training loss is... , where L_LM is the language modeling loss, and λ_floor>0 is the preset weight; this auxiliary loss only takes effect during the training phase and does not change any calculations during the inference phase.

[0017] Furthermore, the competitive normalized attention update in S3 is implemented using a single-round decision, specifically including: S311, splice the old memory and the new candidate along the slot axis: ; S312, Configure a set of learnable decision query matrices Its row number and the final output The number of slots K is consistent; S313, the updated memory is obtained using the following formula: , in , , For learnable projection; the Softmax normalizes along 2K input directions, so that the attention weight of each output slot forms a probability distribution between the K old slots and the K new candidate slots, and the content of the old slots and the content of the new candidates achieve zero-sum competition through this probability distribution.

[0018] An external memory device for a language model includes a memory maintenance module, a memory-attention injection module, an automatic rewriting module, and a two-stage competition module, which are used to maintain the memory matrix and execute the above steps S1, S2, and S3, respectively.

[0019] An electronic device includes a processor and a memory, wherein a computer program in the memory is executed by the processor to implement the above method.

[0020] A computer-readable storage medium thereon, wherein a computer program is executed by a processor to implement the above method.

[0021] Based on the above technical solution, the external memory method for language models based on attention reading and competitive updating, as proposed in this invention, has achieved the following technical effects through practical application: 1. Compared with the "extended context window" scheme, the injection method of the present invention allows each inference to only need to access a constant-size M_c. The computing power of single-token inference is independent of the session history length, there is no quadratic explosion, and it can support arbitrarily long sessions under the same hardware conditions.

[0022] 2. Compared with the “retrieval enhancement generation” scheme, the rewriting and competition of this invention are completed entirely within the forward path of the model by learnable parameters, without the need for embedded models, vector libraries, retrieval pipelines, or reordering models; the learnable query enables “what should be memorized” to be automatically learned from the end-to-end task loss, rather than being determined by manual heuristics; the competition mechanism enables “what should be covered” to be determined by the method itself.

[0023] 3. Compared with the "user-level fine-tuning" scheme, the method of the present invention does not modify the language model ontology parameters, but only maintains the method-side projection parameters that can be shared among users and a very small private M_c per user (typically hundreds of KB to several MB); there is no catastrophic forgetting problem, and it can be easily deployed on the end side. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall process of the external memory method for a language model based on attention reading and competitive updating according to the present invention.

[0025] Figure 2 This is a comparative diagram showing the change in inference overhead with the length of history in Experiment 1 of this invention.

[0026] Figure 3 This is a schematic diagram comparing the memory recall quality with the retrieval enhancement generation scheme in Experiment 2 of this invention.

[0027] Figure 4 This is a schematic diagram showing the comparison of anti-forgetting abilities in Experiment 3 of this invention.

[0028] Figure 5 This is a schematic diagram of the ablation experiment results in Experiment 4 of this invention.

[0029] Figure 6 This is a schematic diagram of the memory capacity K scan results in Experiment 5 of this invention.

[0030] Figure 7 This is a comparative schematic diagram of the three-stage update structure in Experiment Six of this invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and effects of this invention clearer, specific examples are provided below. However, it should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this invention.

[0032] Example 1, Example 1, as Figure 1 As shown, an external memory method for language models based on attention-based reading and competitive updates is proposed. In addition to the conventional forward inference process of the language model, this method configures an additional fixed-capacity circular memory matrix of size K×d, denoted as M_c, where K is the number of slots (typically fixed integers such as 16, 32, 64, 128, 256, 512, 1024, etc.), and d is consistent with the hidden layer dimension of the language model; each session context is independently bound to a private M_c.

[0033] It should be noted that, in this invention, "session" is defined functionally as any consecutive token segment in the dialogue history or input token stream that is designated as a single memory write unit. The session boundary can be defined arbitrarily by dialogue rounds, fixed token windows (e.g., every 512 tokens), time intervals, topics, or semantic segments. Changes in the boundary definition method and segmentation granularity do not affect the implementation and effectiveness of this method. Any implementation that periodically writes historical information into a fixed-capacity memory matrix using learnable compression and updates it competitively, regardless of its specific historical segmentation method, falls within the scope of "session" processing as defined in this invention.

[0034] This method utilizes and maintains the memory matrix through the following three mutually cooperating stages during operation: Phase 1: Memory - Attention Injection; Within each inference session, for several specified layers of the language model, the input hidden state of the current layer is used as the query to read out a memory readout m corresponding to the current token position from M_c in a cross-layer attention manner, and m is injected into the residual stream of the current layer through a deep routing mechanism.

[0035] Specifically, let L be the number of decoder layers in the language model. Define a set of indices for one or more injected layers. In the injected layer In this case, the method performs the following sub-steps: Sub-step A1: For each injected layer Configure a learnable projection matrix unique to this layer. , , Let the input hidden state sequence of this layer at the current inference position be... , where N is the length of the current sequence.

[0036] Sub-step A2: Obtain the memory readout amount of the current layer and current position from M_c using the following formula: in Softmax normalizes along K dimensions (slot axes) so that each query position has a probability distribution across K memory slots.

[0037] Sub-step A3: To improve readout quality, a refinement depth R≥1 is introduced (in this implementation, R=4); iteratively execute the following steps for r=1,2,...,R: Among them each It has independent learnable parameters. Experiment 4's zero-training ablation shows that the long-range memory gain almost completely disappears when the iterative refinement (R=0) is removed. Therefore, this invention takes at least one refinement (R≥1) as an essential technical feature.

[0038] Sub-step A4: Obtain The residual flow's optional source pool is added as the "basic source" of this layer. Let the routable source pool accumulated by this layer along the depth direction be... ,in This represents the memory source injected by this method, and b_0 represents the output of the embedding layer. This represents intermediate residual sources from previous decoded blocks. This method uses learnable pseudo-queries. right Softmax weighting is applied to each source in the dataset: and with As the actual input to subsequent sub-layers of this layer.

[0039] Phase Two: Automatic Rewriting (also known as "Extraction").

[0040] After a session ends and before the next session begins, this method applies a set of learnable rewrite queries to the token representation sequence of the just-ended session, and automatically compresses the information contained in the session into a candidate memory matrix M_new with the same shape as M_c through cross-layer attention.

[0041] Specifically, suppose that the recently concluded conversation obtains a token representation sequence through the language model embedding layer (or a specified intermediate layer). N_c is the number of tokens for this session.

[0042] Sub-step B1: Configure a set of learnable rewrite query matrices The number of rows in M_in is consistent with the number of slots K in M_c. During initialization, each row of M_in is made approximately pairwise orthogonal (e.g., obtained by orthogonal initialization); and during forward propagation, a slot-related positional code (e.g., sine Fourier type code) is added to each row of M_in to break the permutation symmetry of softmax.

[0043] Sub-step B2: Compress this session into a candidate memory matrix using the following formula: in , , For learnable projection. This is the result of compression.

[0044] Sub-step B3 (optional): Introduce extraction depth L_E≥0; for Iterative refinement , , , among which It has independent learnable parameters.

[0045] Sub-step B4 (optional): When C originates from the hidden state of an intermediate layer of the language model, perform RMSNorm normalization on C before feeding it into the extraction stack to avoid the difference in magnitude of hidden states in different layers from damaging the numerical stability of attention.

[0046] Phase Three: Two-Phase Competition.

[0047] This method concatenates the old memory matrix M_c and the candidate matrix M_new along the slot axis into a concatenated matrix of length 2K. Using a set of learnable decision queries as the initial slot states, iteratively performs competitive normalized attention readings and slot-by-slot gated state updates on the concatenated matrix along the slot direction. This allows the old slot content and the new candidate content to compete for K memory slots through a zero-sum contest, thus producing an updated memory matrix. This serves as the initial memory for the next session.

[0048] Specifically, Phase Three will be executed immediately following Phase Two: Sub-step C1: Join the old memory and the new candidate along the slot axis (first dimension): .

[0049] Sub-step C2: Configure a set of learnable decision query matrices Its row number and the final output The number of slots K is consistent. M_judge also uses orthogonal initialization and superimposed slot position encoding to break the symmetry.

[0050] Sub-step C3: With M_judge as the initial slot state s^(0), iteratively perform the following competitive update for i=1,...,T (T=3 in this embodiment): ; ; ; ; in , , The structure is a learnable projection; Softmax_K represents the softmax normalization along the K memory slots; norm_slot represents the weighted average normalization of each slot row to make the row sum equal to 1; GRU is a gated recurrent unit with weights shared across slots and hidden states independent on a slot-by-slot basis. The competitiveness of this structure is reflected in two aspects: First, the attention quality of each of the 2K input slots in P (K old slots + K new candidate slots) is distributed zero-sum along the K memory slots—any old content or new candidate must outperform the other participants in the competition for a memory slot to enter it, thus directly competing with the old content; Second, the GRU gating of each memory slot makes a slot-by-slot trade-off between "preserving the original slot state" and "writing the content that wins the current round of competition". The content of the old slot is only overwritten when the new candidate wins the competition and the gating chooses to write. This method calls this mechanism the "forgetting defense" mechanism.

[0051] Sub-step C4 (optional): To prevent softmax from collapsing into a uniform distribution near the permutation-symmetric solution and losing its binding ability to the specific input, before performing softmax in sub-step C3, for each input column, only the k highest-scoring values ​​in the K output slots are retained, and the logits of the remaining positions are set to zero. Then perform softmax. k is a small integer (typically 1, 2, 4, 8, 16). This modification forces each new candidate information to be bound to at most k slots, disrupting the permutation-symmetric smooth mixing path and allowing the slots to acquire identifiable specialization.

[0052] Sub-step C5 (optional): As an alternative implementation above the output of sub-step C3, a learnable slot-by-slot gating vector can be appended after sub-step C3. Fusion according to the following formula: Where M_judged represents the direct output of sub-step C3.

[0053] Synergistic Relationship of the Three-Stage Approach (Core Innovation of this Invention) This invention does not protect any single isolated stage of Stage 1, Stage 2, or Stage 3, but rather the overall method formed by the three stages working together in the following specific manner: (1) Read from stage one It must come from a fixed capacity matrix M_c that is independent of the session length, thus ensuring constant inference computing power; the cost of this constantness is that the capacity of M_c is limited (K×d). This capacity constraint is the fundamental reason why phases two and three must exist—there must be a mechanism to determine "to accommodate an infinitely long session history with a finite capacity".

[0054] (2) Stage 2 must output a K×d candidate matrix that is strictly consistent with the shape of M_c so that it can directly enter the splicing competition in Stage 3; if Stage 2 outputs a summary sequence of arbitrary variable length, the fixed 2K-softmax competition structure in Stage 3 will not be valid.

[0055] (3) The competitive normalization in stage three must use the 2K slots after splicing M_new from stage two and M_c from the previous time step as the competing participants, and the normalization must constitute a zero-sum competition between the old content and the new content (in this implementation, normalization is carried out along the direction of K memory slots, so that the attention quality of each participant is distributed zero-sum among the K slots); this zero-sum competition structure is the key to the establishment of the "forgetting defense" mechanism (Experiment 6 gives the measured degradation after replacing it with non-competitive update).

[0056] (4) The injection method of "independent projection of each layer + depth routing" in stage one is coupled with the competitive update in stage three: if the injected layer can read information related to the current task from M_c, the loss gradient of that layer will be backpropagated to the decision query M_judge in stage three, thereby guiding it to tend to retain useful slots in the next competition; conversely, if the content of a slot is not selected by any injected layer for a long time, the gradient signal will naturally weaken, and stage three will be more likely to cover it in the competition. This implicit forgetting curve of "use and retain, not use and forget" is automatically generated by the coupling of the three stages.

[0057] (5) The deep routing structure in Phase 1 enables language model layers at different semantic levels to independently decide whether to query M_c. Therefore, M_c can carry information at multiple semantic levels simultaneously during the writing process in Phase 2, without the need for hierarchical classification during writing—hierarchy is automatically completed by the reading process in Phase 1. This "no hierarchical classification during writing, hierarchical classification on demand during reading" working mode is another significant feature that distinguishes this invention from explicit hierarchical memory models.

[0058] It is the interdependence and mutual binding described in (1) to (5) above that makes the three stages together constitute an inseparable whole method; any one of the stages alone cannot achieve the long-range memory effect under constant computing power claimed in this invention.

[0059] Example 2, I. Implementation Environment: A publicly available pre-trained Transformer decoder architecture language model is used as the ontology, denoted by its number of layers L and hidden layer dimension d. This implementation uses a 0.6B parameter scale model with L=28 and d=1024, and a 1.7B parameter scale model with L=28 and d=2048 as examples. This method does not limit the specific parameter scale of the model. The operating environment is a single-card dedicated GPU accelerator (e.g., NVIDIA H100 / H200 / A100, etc.). This method does not limit the specific model, memory capacity, or interconnect topology of the accelerator. This implementation preferably adopts a strategy of freezing the language model ontology parameters and training only the method-side parameters introduced by this method. The freezing strategy itself is not a method step claimed to be protected by this invention, but only serves as an optional environment choice during the engineering implementation phase of this method.

[0060] II. Selection of method-side parameters: In this embodiment, the number of slots in the memory matrix M_c is K=128, therefore That is, approximately 256KB (bf16 precision, 128×1024×2 bytes) in the 0.6B configuration and approximately 512KB (128×2048×2 bytes) in the 1.7B configuration.

[0061] In stage one, the injected layers are selected as several layers uniformly distributed in {1,2,...,L} (typically selected every...). (Injected once per layer); each layer is configured independently. , , Three sets of d×d projections.

[0062] The refinement depth of stage one is R=4; to ensure that the original forward behavior of the language model is not disturbed during zero initialization, the refinement depth of each refinement layer is... Initialize with zero ("ReZero" style).

[0063] The extraction depth in stage two is L_E=4; The row vectors are initialized orthogonally and the sine Fourier positional encoding of the slot dimension is superimposed during forward propagation to break the symmetry.

[0064] The input source for stage two uses an intermediate layer of the language model (e.g., The hidden state of ) is normalized by RMSNorm and then sent to the extraction stack.

[0065] Phase 3 Judgment Inquiry Similarly, it uses orthogonal initialization + slot position encoding; , , The d×d projection is initialized for regular Xavier; slot contention softmax is normalized along the direction of K=128 slots and weighted average normalization is performed for each slot; the number of iterations T=3; the hidden state dimension of the GRU gated recurrent unit is d.

[0066] In Phase 3, the optional sparse top-k module takes k=8.

[0067] Phase 1 deep routing pseudo-query The method initializes the model to zero and sets an initial bias for its bias term to ensure that the model's forward output is "bit-level consistent" at training step 0, thus avoiding initialization noise disturbance.

[0068] III. Step-by-step execution flow of the method, let the current session be the t-th session; the memory matrix of the previous time step is denoted as... This implementation method is carried out according to the following steps: Step S1 (Inference Phase – Phase 1 Execution): The current token sequence within this session is sequentially fed into the language model; in each injected layer Place: Calculation ;by , , and calculate ;right Perform R-fold refinement; Added to the deep routing source pool of this layer as the base source b_{-1} and according to learnable pseudo-queries Perform softmax weighting to obtain This serves as the input for subsequent sub-layers. The loop continues until the current session ends.

[0069] Step S2 (Inter-session phase – Phase 2 execution): After the current session ends, retrieve the token representation sequence C corresponding to this session; if C comes from the intermediate layer, perform RMSNorm normalization on C; use M_in as the query and C as the key to perform a single cross-layer attention compression to obtain... ;right Perform L_E=4 extraction and refinement steps to obtain candidate memories. .

[0070] Step S3 (Inter-session phase – Phase 3 execution): ... spliced ​​with M_new along the slot axis as With M_judge as the initial slot state, perform T=3 iterations of slot competition update on P—each iteration: calculate the slot-input attention score; if the sparse top-k module is enabled, only the k slots with the highest scores are retained for each input column (the rest are set to zero). ); perform softmax along the K slot directions and normalize by weighted average for each slot; read the weighted values ​​and update the slot state using GRU – after iteration, obtain the RMSnorm result. ;Will This will serve as the memory matrix used when executing step S1 in the next session.

[0071] Step S4 (Multi-session context deployment): For each session context Independently maintain its private The method-side parameters involved in steps S1 to S3 are shared across all session contexts. Stored anywhere on the client side, in the user's encrypted cloud, or on a centralized server, and loaded by the u index on each inference call.

[0072] IV. Training methods for method-side parameters In this embodiment, all method-side learnable parameters (readout projections of each injected layer and refinement layers, depth-routing pseudo-queries) The rewritten query M_in, projection extraction and refinement layer extraction, and decision query M_judge and decision projection are jointly trained under the condition that the language model ontology parameters are completely frozen. The training data is a multi-conversation dialogue chain (512 tokens per session); during training, k=3 consecutive sessions are unfolded in the time-truncated backpropagation (TBPTT) manner: memory writing of phase two / phase three is performed for each session, and the cross-entropy loss of language modeling per token is calculated for each session within the window under the condition of phase one reading (the loss of the evidence session itself is masked, and the loss of filling tokens is not counted), where the loss of callback tokens that depend on cross-conversation history information is weighted by a factor of 1+5.0. The optimizer is... The learning rate was 1e-4 (cosine annealing after 200 linear warm-up steps), and the training took 1000 steps. The batch size was 4, the gradient accumulation was 2 (effective batch size was 8), and the gradient norm was clipped to 1.0. During training, M_c was randomly set to zero with a probability of 0.10 (memory dropout), and the context session was randomly dropped with a probability of 0.05 (context dropout).

[0073] Total training loss is Where L_LM is the weighted language modeling loss mentioned above; L_floor is the auxiliary loss representing the lower bound of the proportion of deep route memory.

[0074] in For the set of all injected layers, For the first Layer depth routing assigns softmax weights to memory source b_{-1} The average value across all token positions in the current training batch; in this implementation, τ = 0.10 and λ_floor = 0.5. This auxiliary loss belongs to the load balancing type: without this, deep routing will collapse the memory source weights to approximately 0.001 in the early stages of training (similar to the expert collapse phenomenon in hybrid expert models), causing the parameters of stage two / stage three to fail to obtain downstream gradients and thus become untrainable (see Experiment 4 for actual ablation testing); with this added, the routing is forced to maintain the minimum sampling amount of the memory source, allowing the method to train successfully end-to-end. This auxiliary loss only takes effect during the training phase; the method calculation during the inference phase is completely unrelated to it.

[0075] With a 0.6B configuration, the method side has approximately 41.5M trainable parameters (about 6% of the ontology parameters); training is performed on a single NVIDIA H100 card; random seeds are taken as {1,2,3,4}, and results are reported as mean ± standard deviation of multiple seeds.

[0076] V. Several Alternative Embodiments: This section lists several alternative embodiments, all of which fall within the protection scope of this invention: Regarding the injection location in Phase 1: The set of indices for the injected layers can be selected as injecting all layers, injecting only a few specified layers, or dynamically determining the injection layer based on the learnable layer selection gate.

[0077] Regarding the input source for stage two: C can be the output of the embedding layer, the hidden state of any specified intermediate layer, or the concatenation or weighted sum of multiple hidden states.

[0078] Regarding the slot update strategy in Phase 3: a pure competitive mode (directly using the output of C3 as M_c^(t)) or a gated mode (gating and merging C5 with the old memory) can be adopted. Furthermore, iterative slot competitive update can also be simplified to the single-round 2K-dimensional softmax decision described above. The Softmax normalizes along 2K input directions, causing the attention weights of each output slot to form a probability distribution among the K old slots and K new candidate slots. This single-round variant structure is simpler (no iteration, no gated loop unit, and the number of parameters is about one-third of the iterative implementation). The measured performance of training from scratch is lower than that of the iterative implementation described in claim 6 (see Experiment 6), but it still significantly exceeds all retrieval augmentation baselines, and can be used as an alternative implementation in scenarios where computing power or parameter budget is limited.

[0079] Regarding sparse top-k correction: it can be applied only to stage one, only to stage three, or both; typical values ​​for k are 1, 2, 4, 8, and 16.

[0080] Regarding the execution timing of Phase Two / Phase Three: The generation of the candidate memory matrix (Phase Two) and the competitive update (Phase Three) can be executed synchronously immediately after the end of a session, or they can be executed delayed, asynchronously, or in batches for multiple sessions; as long as the updated memory matrix is ​​written before it is subsequently read and used, changes in the execution timing and execution method do not affect the implementation and effect of this method, and are all implementation methods of this invention.

[0081] Regarding the number of slots K: it can be selected within the range of {16, 32, 64, 128, 256, 512, 1024, ...}.

[0082] Regarding numerical precision and storage format: M_c can be stored with any numerical precision such as bf16, fp16, int8, int4, etc.; the storage location can be on the user terminal, encrypted cloud, trusted execution environment, or centralized server.

[0083] VI. Applicable Fields Preferred application areas for this method include, but are not limited to: intelligent dialogue assistants, role-playing and companion AI, customer service dialogue systems, autonomous coding and research agents, medical dialogue systems (maintaining memory by patient), educational tutoring systems (maintaining memory by student), cognitive assistance systems for people with cognitive impairments, and any application scenario that requires providing long-term conversational memory capabilities without modifying the weights of the base model.

[0084] VII. Experimental Verification To verify the technical effectiveness of this invention, a systematic experiment was conducted on the embodiments described in the "Specific Implementation" section on the publicly available long-range conversational memory benchmark, LongMemEval-S. Unless otherwise specified, all experiments used pre-trained language models Qwen3-0.6B (L=28, d=1024) and Qwen3-1.7B (L=28, d=2048) with completely frozen parameters as the ontology; the default memory matrix slot number K=128, readout refinement depth R=4, extraction depth L_E=4, and iterative slot contention update as described in Stage 3 (T=3). As a control, the single-round 2K-softmax decision variant was also trained and evaluated from scratch with the exact same formula (see Experiment 6). The method-side parameters were obtained by training for 1000 steps on the LongMemEval-S training partition (450 session chains) using end-to-end language modeling loss (approximately 41.5M trainable parameters under 0.6B configuration, about 6% of the ontology parameters), and evaluated on the validation partition (50 session chains, approximately 50 sessions per chain, 512 tokens per session, i.e., approximately 25,000 tokens of cross-session history). The evaluation metric is "callback cross-entropy," which is the average cross-entropy (in NATs, lower is better) of key tokens in the callback session that must rely on cross-session history information to be correctly generated; denoted as Δ = (callback cross-entropy of the memoryless baseline) - (callback cross-entropy of the tested method), the larger the Δ, the greater the improvement in generation quality from long-range memory. As an intuitive supplement to cross-entropy, some experiments also report "callback token greedy exact matching rate," which is the exact matching ratio of the argmax prediction of the callback token at each position with the reference token (higher is better). All comparison and ablation units trained from scratch used the exact same training data, training steps, optimizer, and hyperparameters as the full method (see the "Training Method-Side Parameters" section). All baseline schemes used the same and equally frozen language model ontology, and their adjustable settings (RAG retrieval types and top-k, LoRA learning rate and update protocol) were scanned and the combination most favorable to the baseline was selected as the main comparison. All raw logs, checkpoints, and configuration files from all experiments have been archived. The hardware environment was a single NVIDIA H100 GPU.

[0085] Experiment 1: Quantitative comparison of inference overhead with history length (compared with the "extended context window" scheme) Experimental setup: The baseline is the same Qwen3-0.6B model where all history is concatenated into the context window using the original tokens; this invention writes the history into a fixed-capacity memory matrix M_c for each session with 512 tokens, and only carries the current session (512 tokens) and M_c during inference.

[0086] The history length H increased progressively from 1,024 to 32,768 tokens (all real conversation tokens). Pre-fill latency, peak memory usage for a single inference attempt, and the size of persistent state required to maintain the session were measured (baseline was KV cache; this invention uses the current session KV cache + M_c). Each measurement was repeated 5 times, and the median was taken. The results are shown in Table 1. Figure 2 As shown.

[0087] Table 1 shows the variation of inference overhead with history length H (latency in ms, video memory in GiB, and storage in MiB).

[0088] Experimental Results: The pre-fill latency of the baseline scheme increases superlinearly with H (from 19.2ms to 718.9ms when H increases from 1,024 to 32,768, approximately 37 times), the peak memory usage increases from 1.35 GiB to 5.54 GiB, and the KV cache increases linearly from 168 MiB to 3,640 MiB. The pre-fill latency (approximately 22.8ms), peak memory usage (1.63 GiB), and persistent state (approximately 56 MiB) of this invention remain constant across all H values, completely independent of the history length; the one-time write overhead at the end of the session is also constant (approximately 8.2ms / session). Each user only needs to store the M_c matrix itself (256KB at bf16 precision) for long-term storage, while the baseline at H=32,768 requires approximately 3.6GB of KV cache alone, a difference of approximately four orders of magnitude. These results directly verify the technical effect 1 described in the "Summary of the Invention".

[0089] Experiment 2: Quantitative Comparison of Memory Recall Quality (Compared with the "Retrieval Enhancement Generation (RAG)" Scheme) Experimental setup: The RAG baseline stores all historical sessions in the retrieval database. During inference, the top-k historical sessions are retrieved using callback sessions as queries, and the original text is then incorporated into the context. The retrieval system employs (a) BM25 lexical retrieval, (b) dense vector retrieval (all-MiniLM-L6-v2 embedding model), and an additional (c) "ideal retrieval (oracle)" reference—directly incorporating manually annotated evidence sessions into the context. This reference only represents the upper limit of the performance of the "retrieval-original text concatenation" RAG setup used in this experiment when the retrieval system is idealized (100% precision), and is not the theoretical upper limit of all retrieval enhancement schemes. The language model ontology of RAG is the same as that of this invention and is also frozen. The callback cross-entropy improvement Δ is evaluated on the same validation set. The results are shown in Table 2 and... Figure 3 As shown.

[0090] Table 2 compares the callback cross-entropy improvement Δ with the RAG scheme (nats, the larger the better).

[0091] Experimental Results: At a scale of 0.6B, the highest deployable RAG variant achieved a Δ = +0.048 (top-3 in dense retrieval), and even the "ideal retrieval" reference only achieved +0.111. This invention achieved a Δ = +1.290 ± 0.501, approximately 27 times that of the optimal actual RAG variant and 12 times that of the ideal retrieval reference. At a scale of 1.7B, this invention (+0.973 ± 0.106) also significantly outperformed the ideal retrieval reference (+0.388). Measured by the greedy exact match rate, this invention increased the positional hit rate of callback tokens from 32.5% (without memory) to 48.4% (with memory enabled). Meanwhile, RAG requires hundreds to thousands of additional retrieval tokens for each inference and relies on external infrastructure such as embedded models and vector libraries; this invention requires zero additional tokens and has no external retrieval components. These results directly verify the technical effect 2 described in the "Invention Content".

[0092] Experiment 3: Quantitative comparison of anti-forgetting ability (compared with the "user-level fine-tuning" scheme) Experimental setup: (a) On the present invention side: competitive phase two / phase three updates were performed on each of the 50 chains in the validation set for each session. The callback cross-entropy was measured once for each session written (the callback token depends on information from earlier sessions in the chain, so this metric simultaneously measures the combined effect of "new information adoption" and "old information retention"); (b) On the fine-tuning baseline side: Simulating existing "user-based fine-tuning" techniques, each chain (considered as the history of one user) was continuously fine-tuned on the same Qwen3-0.6B ontology using LoRA (rank 16, applied to all attention and feedforward projection matrices), and various update protocols and learning rates were swept through—including sequential session-based updates and "joint offline" updates (the entire chain history was shuffled and sampled uniformly with the same total number of steps, representing the strongest per-user offline fine-tuning setting). The main comparison used a variant of joint, lr=2e-05. Simultaneous measurements were taken of (i) callback cross-entropy and (ii) cross-entropy on general text outside the fine-tuning domain (fineweb-edu English corpus), which is completely different from the fine-tuning domain. The results are shown in Table 3. Figure 4 As shown.

[0093] Table 3 Comparison of anti-forgetting ability and renewal cost (Qwen3-0.6B)

[0094] Experimental Results: During the continuous competitive update process of approximately 50 sessions across the entire chain, the callback cross-entropy of this invention continuously decreased, improving monotonic accumulation to +1.739 nts (average across 50 chains, decreasing from 4.009 to 2.270). The written information was retained even after dozens of subsequent competitive updates. Furthermore, due to the complete freezing of ontology parameters, the drift of out-of-domain general capabilities was strictly zero. The user-level LoRA fine-tuning baseline (main comparison: joint, lr=2e-05) showed a callback cross-entropy change of +0.689 nts after the entire chain update, and an increase in out-of-domain general text cross-entropy of +0.039 nts, demonstrating the dual damage of continuous updates to early information and general capabilities. Each user needs to store 38.5 MB of adapter parameters—approximately 154 times the 256 KB memory matrix per user of this invention—and requires multiple backpropagation steps per session (this invention only requires one forward write). These results directly verify the technical effect 3 described in the "Invention Content".

[0095] Experiment 4: Three-stage synergy and ablation experiment of key components Experimental Setup: To verify the assertion in the section "Synergistic Relationship of the Three-Stage Method" that "the three stages constitute an indivisible whole," and the necessity of key design choices within Stage 1, this experiment was conducted in two layers. The first layer was ablation training from scratch: Under the same training formula, data, and number of steps as the complete method, (a) the complete method was trained; (b) a readout refinement depth R=0 (removing the iterative readout of Stage 1); (c) the depth routing α-floor auxiliary loss was removed; and (d) the competitive update of Stage 3 was replaced with a "covering" update (the candidate matrix of each new session directly covers the old memory, with no 2K-dimensional zero-sum competition, training from scratch). The second layer was stage ablation during evaluation: At the checkpoints where the complete method was trained, stages were progressively closed during inference, sharing the identical Stage 2 extraction output for each session. The results are shown in Table 4. Figure 5 As shown.

[0096] Table 4 shows the ablation experiment: the improvement in cross-entropy during the callback experiment was Δ (nats; 0.6B).

[0097] Experimental Results: The full method achieved a significant improvement of +1.323±0.530 (n=4); after training from zero and removing readout refinement, the improvement collapsed to +0.049±0.025 (n=4); after removing the deep routing α-floor, the improvement collapsed to +0.060±0.071 (n=4); and replacing stage 3 with overlay updates and training from zero only achieved +0.937 (2 seeds), all lower than the full method. During evaluation, stage ablation exhibited the same "1+1+1>3" structure: only stage 1 (+0.049) and the two non-competitive variants of stage 1+2 (overlay -0.073, mean -0.074) were close to or lower than the memoryless baseline; only using all three stages achieved significant improvements of +1.290 (0.6B) and +0.973 (1.7B).

[0098] The failure of "Phase 1 + Phase 2 (overlay)" confirms the assertion of "synergistic relationship of the three-phase method" (3) - without zero-sum competition between old and new content, the "forgetting defense" mechanism is not valid; the failure of "Phase 1 only" confirms the assertion (1) - without Phase 2 / 3, the fixed capacity matrix does not contain any historical information that can be read. The above results directly support the necessity of claim 1 to protect the three phases as a whole.

[0099] Experiment 5: K-scan of memory capacity Experimental setup: Under the same training formula as the complete method, only the number of slots K in the memory matrix was changed. Training was performed from scratch with K∈{32,128,512}, and evaluation was conducted on the same validation set to verify the feasibility of the specification's statement that "K can be selected within the range of {16,32,64,128,256,512,1024,…}", and to quantify the impact of capacity on memory performance. Results are shown in Table 5. Figure 6 As shown.

[0100] Table 5 shows the K-scan of memory capacity (Qwen3-0.6B, trained from scratch).

[0101] Experimental Results: Within the scanned range of K ∈ {32, 128, 512}, the present invention achieves positive callback cross-entropy improvement, indicating that fixed-capacity external memory can be implemented under different slot budgets; the default configuration K=128 (approximately 256KB / user) achieves a good trade-off between performance and storage. This result supports the implementation description regarding the range of K values ​​in the specification.

[0102] Experiment 6: Comparison of Stage 3 Update Structures (with the closest existing memory update structure) Experimental Setup: To directly answer the question, "Is the competitive decision update of Phase 3 of this invention obvious based on existing memory update structures such as relational memory / gated memory?", this experiment uses the update structure of Phase 3 as the only variable: Phase 1, Phase 2, training recipe, data, and number of steps are all kept consistent with the complete method, except that Phase 3 is replaced with the following structures and then trained from scratch and evaluated: (a) the iterative slot competitive update (this invention, T=3, 4 seeds); (b) the single-round 2K-softmax competitive decision (a simplified variant of this invention, 4 seeds); (c) "Relational memory kernel (Rela (d) Gated replacement update – each slot uses a sigmoid gate to linearly interpolate between the old slot and the new candidate (g·new + (1-g)·old), without any attention competition; (e) Overlay update – the new candidate directly overwrites the old memory; (f) Mean pooling update – the mean of all historical candidates (replaced during evaluation). The results are shown in Table 6. Figure 7 As shown.

[0103] Table 6 shows a comparison of the phase 3 update structure: the callback cross-entropy improvement Δ (nats; Qwen3-0.6B, except for (f), all were trained from zero, and the other components were exactly the same as the training formula).

[0104] Experimental results: (1) The iterative slot competition update achieved the best performance of +1.323±0.530 among all tested update structures. Its zero-sum competition along the slot direction forces the K memory slots to explicitly compete between old content and new candidates, and makes each slot specialize in different information in history (avoiding the symmetric collapse of "all slots converging to the same mean"), which is the structural reason for achieving this effect. (2) The single-round 2K-softmax simplified variant achieved +0.724±0.317, which is about 45% lower than the iterative implementation, but still exceeds all RAG variants in Experiment 2 (including the ideal retrieval reference +0.111), and can be used as a lightweight alternative implementation. (3) The three known update structures (relational memory kernel self-attention +0.846, gated replacement +0.882, and covering +0.937) can also achieve positive improvement on the Phase 1 / Phase 2 architecture of this invention - which itself further confirms the contribution of Phase 1 and Phase 2 (compared to Experiment 4: the overall gain is almost zero after removing readout refinement or deep routing auxiliary loss) - but all are about 30%~40% lower than the iterative slot competition update of claim 6. (4) The necessity of competitive decision during the inference period is directly proved by the ablation during the evaluation of Experiment 4: at the same well-trained complete method checkpoint, simply replacing Phase 3 with covering or mean (without retraining) will cause the improvement to drop sharply from +1.290 to -0.073 / -0.074, indicating that the system after training convergence actually relies on competitive decision to maintain cross-session memory. In summary, Phase 3, "Iterative slot zero-sum competition driven by learnable decision queries," has proven technical advantages over well-known self-attention updates, gated replacements, and overlay updates. This advantage is based on the overall synergy of Phase 1, Phase 2, and Phase 3, and a simple replacement of any well-known structure cannot achieve the effect of the complete method.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A method for external memory of a language model based on attention-based reading and competitive updating, characterized in that, The method specifically includes the following steps: S1, Memory-Attention Injection: In each inference session, for one or more specified layers of the language model, the input hidden state of that layer is used as a query. A memory readout is read from the fixed-capacity memory matrix in a cross-layer attention manner. The memory readout is refined at least once using the memory matrix as the key. The refined memory readout is then injected into the residual stream of that layer through a deep routing mechanism. S2, Automatic Rewriting: After a session ends and before the memory matrix is ​​read by a subsequent session, a set of learnable rewriting queries is applied to the token representation sequence of the just-ended session. Through cross-layer attention, the information contained in the session is compressed into a candidate memory matrix with the same shape as the memory matrix. S3, a two-stage competition, concatenates the old memory matrix and the candidate matrix along the slot dimension into a concatenated matrix of length 2K. A set of learnable decision queries is applied to the concatenated matrix, and competitive attention updates based on softmax normalization are performed. The old slot content and the new candidate content form a zero-sum competition through softmax normalization. The competition result determines the content of each slot in the updated memory matrix, and the updated memory matrix is ​​produced as the initial memory for the next session.

2. The method for external memory of a language model based on attention-based reading and competitive updating as described in claim 1, characterized in that, The language model has L decoder layers, and one or more sets of indices are set for the injected layers. The memory-attention injection in S1 specifically refers to: S11, for each injected layer Configure a learnable projection matrix unique to this layer. Let the input hidden state sequence of this layer at the current inference position be... Where N is the length of the current sequence; S12, obtain the memory readout amount of the current layer and current position from the fixed-capacity memory matrix, specifically as follows: ;in For the current input hidden state sequence of this layer, Softmax normalizes it along the K-dimensional axis; S13, will be obtained As the base source for this layer, it is added to the optional source pool of the residual flow. Let the routable source pool accumulated by this layer along the depth direction be: ， in This indicates the memory source injected by this method, and b_0 represents the output of the embedding layer. This indicates intermediate residual sources from previous decoded blocks; With learnable pseudo queries right Softmax weighting is applied to each source in the dataset: ， ； and with As the actual input to subsequent sub-layers of this layer.

3. The method for external memory of a language model based on attention-based reading and competitive updating as described in claim 2, characterized in that, S1 further includes: introducing a refinement depth R≥1, and iteratively performing the following steps for r=1,2,...,R: , among which It has independent learnable parameters.

4. The method for external memory of a language model based on attention-based reading and competitive updating as described in claim 1, characterized in that, S2 specifically includes: S21, Configure a set of learnable rewrite query matrices The number of rows is consistent with the number of slots K in M_c; each row of M_in is initialized orthogonally, and the slot position encoding is superimposed on each row during forward propagation to break the permutation symmetry; S22, compressing this session into a candidate memory matrix specifically is as follows: ， in This is the token representation sequence for this session. For learnable projection, This is the result of compression; S23, output E as M_new.

5. The method for external memory of a language model based on attention-based reading and competitive updating as described in claim 4, characterized in that, S2 further includes: introducing an extraction depth L_E≥0, for Iterative execution: ， ， ， Among them each It has independent learnable parameters.

6. The method for external memory of a language model based on attention-based reading and competitive updating as described in claim 1, characterized in that, S3 specifically includes: S31, splice the old memory and the new candidate along the slot axis: ; S32, Configure a set of learnable decision query matrices Its row number and the final output The number of slots K is consistent; Orthogonal initialization and overlay of slot position codes; S33, with Initial slot state For i=1,...,T, perform the following competitive update, where T is the preset number of iterations: ; ; ; ,in , , For learnable projection; Softmax_K represents softmax normalization along the K memory slots, so that the attention quality of each of the 2K input slots (K old slots and K new candidate slots) in the concatenation matrix P is zero-sum distributed among the K memory slots; norm_slot represents weighted average normalization of each memory slot row so that the row sum is 1; GRU is a gated recurrent unit with weights shared across slots and hidden states independent per slot, realizing slot-by-slot retain / write gating.

7. The method for external memory of a language model based on attention-based reading and competitive updating as described in claim 6, characterized in that, S3 further includes: before performing Softmax normalization, for each input column, only the k highest-scoring columns are retained in the K output slots, and the logits of the remaining positions are set to zero. Then perform Softmax; where k is a preset small integer; After performing Softmax normalization, a learnable slot-by-slot gate vector is appended. , Gating fusion is performed as follows: , where M_judged represents the output after the S33 iteration ends.

8. The method for external memory of a language model based on attention-based reading and competitive updating as described in claim 2, characterized in that, The set of injected layer indices can be any one of the following: injecting all layers, injecting only a few specified layers, or dynamically determining the injected layers according to a learnable layer selection gate.

9. The method for external memory of a language model based on attention-based reading and competitive updating as described in claim 1, characterized in that, The learnable parameters introduced in this method are obtained by training with end-to-end language modeling loss under the condition that the ontology parameters of the language model are frozen; and the total training loss also includes an auxiliary loss with a lower bound on the proportion of deep routing memory. ； in For the set of all injected layers, For the first Layer depth routing is assigned to memory source softmax weights The average value across all token positions in the current training batch, τ∈(0,1) is the preset lower threshold, ReLU(x)=max(x,0); the total training loss is... Where L_LM is the language modeling loss, The weights are preset; this auxiliary loss only takes effect during the training phase and does not change any calculations during the inference phase.

10. The method for external memory of a language model based on attention-based reading and competitive updating as described in claim 1, characterized in that, The competitive normalized attention update in S3 is implemented using a single round of decision-making, specifically including: S311, splice the old memory and the new candidate along the slot axis: ; S312, Configure a set of learnable decision query matrices Its row number and the final output The number of slots K is consistent; S313, the updated memory is obtained using the following formula: ， in , , For learnable projection; the Softmax normalizes along 2K input directions, so that the attention weight of each output slot forms a probability distribution between the K old slots and the K new candidate slots, and the content of the old slots and the content of the new candidates achieve zero-sum competition through this probability distribution.

11. An external memory device for a language model based on attention-based reading and competitive updating, characterized in that, include: The memory maintenance module is used to maintain a memory matrix of fixed capacity for each session context, which is independent of the session length and history length. A memory-attention injection module for performing step S1 in the method as described in any one of claims 1 to 10; An automatic rewriting module is configured to perform step S2 in the method as described in any one of claims 1 to 10; A two-stage competition module for performing step S3 in the method as described in any one of claims 1 to 10.

12. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program, which, when executed by the processor, implements the language model external memory method based on attention readout and competitive update as described in any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the language model external memory method based on attention readout and competitive update as described in any one of claims 1 to 10.

Images