Robust enhancement method and system for long context processing based on dynamic adjustment of neurons

Abstract

The application discloses a long context processing robust enhancement method and system based on neuron dynamic adjustment, and belongs to the technical field of text sequence processing, and comprises the following steps: embedding a nerve regulation unit in each Transform layer of a large language model used for a long context processing task; and the nerve regulation unit sequentially performs the following steps: dynamically activating abnormal perception, that is, adaptively calculating an abnormal detection threshold of a current layer according to activation value statistical characteristics of the current layer; activating a hard-gate truncation, that is, generating a binary hard mask according to the abnormal detection threshold, and the binary hard mask realizes deterministic isolation of abnormal components in the activation value while keeping normal components unchanged; and performing hierarchical response compression, that is, performing learnable dimension-level linear compression on the isolated abnormal components, re-integrating the compressed abnormal components and the normal components through gate fusion, and outputting an activation tensor after suppressing the abnormality. The application can prevent rank collapse and gradient oscillation, and enhance the robustness of long context task execution.

Description

Technical Field

[0001] This invention belongs to the field of text sequence processing technology, specifically relating to a robust enhancement method and system for long context processing based on dynamic neuron regulation. Background Technology

[0002] With the rapid development of Large Language Models (LLMs), the ability to handle long contexts has become a key indicator for evaluating the practicality of models. In practical applications, tasks such as long text reading comprehension, multi-step reasoning in mathematical word problems, long document summarization generation, multi-turn dialogue context understanding, and cross-line dependency handling in code generation all require models to be able to efficiently handle long sequences of tens of thousands or even hundreds of thousands of tokens.

[0003] However, existing large language models face a key technical bottleneck when handling long context tasks: the problem of activation anomaly accumulation during long sequence processing. Specifically, this manifests as follows: (1) Generation and amplification of activation anomalous components: There are a small number of numerically anomalous dimensions (i.e., anomalous weights) in the weight matrix of the pre-trained large language model. When processing long context sequences, as the sequence length increases, these anomalous weights are repeatedly activated, generating activation anomalous components with amplitudes far exceeding the normal range, and these anomalous components are continuously accumulated and amplified during the interlayer propagation process.

[0004] (2) Representing the rank collapse problem: The accumulated activation anomalies dominate the covariance structure of the feature matrix, causing the effective rank of the feature matrix to drop sharply. In long context tasks, this rank collapse effect is more severe: the longer the sequence, the more activation anomalies accumulate, the fewer the effective representation dimensions of the model, and ultimately the degradation of the model's expressive ability, making it unable to distinguish information at different positions in long sequences.

[0005] (3) Gradient oscillation and training instability: The activation of anomalous components amplifies the gradient through the chain rule, resulting in an abnormally large increase in the gradient update amount in the direction of anomalous weights, which further exacerbates the weight anomalousness and forms a positive feedback loop of "activation anomalousness → gradient anomalousness → aggravated weight anomalousness". In the fine-tuning process of long context tasks, this positive feedback loop leads to severe loss oscillation, slow convergence, or even complete divergence.

[0006] Existing technologies have significant shortcomings in addressing the above-mentioned problems: (1) Static quantization methods (such as SmoothQuant and LLM-int8): Based on fixed thresholds in offline statistics, they cannot adapt to the dynamic changes in activation anomalies with sequence length during long context processing, and have limited suppression effect on the cumulative amplified activation anomaly components.

[0007] (2) Global normalization methods (such as RMSnorm and PowerNorm): Adjust the activation statistics at the distribution level, but cannot accurately identify and suppress sparse and transient abnormal activation spikes that occur in forward propagation, especially in the latter half of long sequences, where the frequency of abnormal spikes increases significantly.

[0008] (3) Existing efficient parameter fine-tuning methods (such as LoRA and DoRA): Assuming that the pre-trained weights are in a benign state, no dynamic adjustment mechanism is designed for the abnormal accumulation of activation in long context scenarios, so the above positive feedback loop cannot be broken.

[0009] Therefore, there is an urgent need for a robust enhancement scheme for long context processing that can detect the accumulation of activation anomalies in real time, dynamically suppress abnormal components, and maintain the complete flow of normal information. Summary of the Invention

[0010] In view of the above, the present invention provides a robust enhancement method and system for long context processing based on dynamic neuron regulation. By drawing on the threshold-driven firing and adaptive regulation mechanism of biological neurons, it dynamically senses and suppresses abnormal activation components during long context processing, prevents rank collapse and gradient oscillation, and enhances the robustness of long context task execution.

[0011] To achieve the above-mentioned objectives, an embodiment provides a robust enhancement method for long context processing based on dynamic neuron modulation, comprising the following steps: A neural modulation unit is embedded in each Transformer layer of the large language model used for long context processing tasks. The neural modulation unit sequentially performs dynamic activation of anomaly perception, activation of hard gating truncation, and hierarchical response compression steps. The dynamic activation anomaly perception includes: adaptively calculating the anomaly detection threshold of the current Transformer layer based on the statistical characteristics of the activation value of the current Transformer layer; The activation hard gating cutoff includes: generating a binary hard mask based on an anomaly detection threshold. The binary hard mask enables deterministic isolation of the abnormal components in the activation values, while keeping the normal components in the normal activation values ​​unchanged. The hierarchical response compression includes: performing learnable dimensional linear compression on the isolated anomalous components, reintegrating the compressed anomalous components with the identity-passing normal components through gated fusion, and outputting an activation tensor after suppressing the anomalous components. This effectively suppresses activation anomalies caused by the accumulation of long sequences in long context processing tasks, thereby enhancing the robustness of task execution.

[0012] Preferably, the step of adaptively calculating the anomaly detection threshold of the current Transformer layer based on the statistical characteristics of the activation values ​​of the current Transformer layer includes: For the activation tensor of the current Transformer layer ,in For batch size, For sequence length (in long context tasks, (It can reach tens of thousands or even hundreds of thousands) For feature dimension, If it is a tensor space, then the anomaly detection threshold is... Calculated using the following dual-benchmark strategy: in, The norm of the current batch activation value The 90th percentile is used as an adaptive baseline, which is adaptively adjusted according to the dynamic changes of the long context sequence, and can capture the cumulative effect of activation anomalies as the sequence length increases. The median of the norm of the current batch of activation values. 1.0 serves as an absolute benchmark to prevent the undetected activation components with small but still anomalous amplitudes in long sequences; 1.0 is the lower limit of numerical stability. To find the maximum value.

[0013] Preferably, generating a binary hard mask based on an anomaly detection threshold includes: For activation tensor H Each element in Generate the corresponding binary mask. : in This is an indicator function that determines when the activation value norm exceeds a threshold. The time mask value is 1 if it is not 0 otherwise. subscript These are batch index, sequence long position index, and feature dimension index, respectively.

[0014] This binary hard mask has the following characteristics: (a) Determinism: It is directly determined based on a statistical threshold, without introducing randomness, ensuring the consistency and reproducibility of long context processing; (b) Sparsity: The positions with a mask value of 1 account for only a very small proportion of the activation tensor (usually less than 0.5%), even in long sequences ( In the scenario of (=32768), the computational overhead is still negligible; (c) Gradient pass-through: This binary hard mask is separated from the computation graph during backpropagation and is only used as a constant coefficient for gradient path selection. The computation graph refers to the directed acyclic graph in deep learning frameworks (such as PyTorch and TensorFlow) that records all forward propagation operations (such as matrix multiplication, activation functions, element-wise operations, etc.) and their dependencies, used to automatically calculate gradients during backpropagation. The process of selecting gradient paths based on this conventional coefficient is as follows: abnormal positions participate in gradient updates after compression transformation, while normal positions maintain the original gradient path, ensuring that the gradients of normal activation values ​​carrying long contextual key information are not disturbed.

[0015] Preferably, the isolated anomalous components are subjected to learnable dimensional linear compression, including: For abnormal components that are marked as abnormal by the mask Apply the following compression transformation: in, , For feature dimension, The compression parameter is independently learnable for each feature dimension. This compression parameter is jointly optimized with other model parameters during model training for long-context tasks, enabling the model to adaptively learn the optimal suppression strength for outlier components on different feature dimensions. The sigmoid function constrains the compression coefficient to the interval (0,1). This indicates element-wise multiplication. These are the compressed anomalous components; For normal components that are masked and marked as normal Perform constant pass-through, that is, constant pass-through of the normal component. This ensures that the vast majority of normal activation values ​​carrying key semantic information in long contexts are not distorted in any way.

[0016] Preferably, the compressed anomalous components are reintegrated with the normally passed components through gating fusion, including: Where 1 represents a tensor of all 1s. The activation value output by the neural regulatory unit after the inhibition of abnormality. The original activation values ​​are replaced and fed into subsequent sub-layers of the current Transformer layer (such as attention layers or feedforward networks). It is a binary mask.

[0017] Preferably, the neural modulation unit dynamically updates the anomaly detection threshold in training mode and uses the final anomaly detection threshold saved at the end of training in inference mode, and the learnable compression parameters in the hierarchical response compression are also used. Joint optimization with other parameters of large language models during model training for long context processing tasks.

[0018] Preferably, the long context processing task includes at least one of the following: long text reading comprehension, multi-step reasoning for mathematical word problems, long document summarization generation, multi-turn dialogue context understanding, and cross-line dependency processing in code generation.

[0019] To achieve the above-mentioned objectives, the embodiments also provide a robust enhancement system for long context processing based on dynamic neuron modulation, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned robust enhancement method for long context processing based on dynamic neuron modulation.

[0020] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-mentioned robust enhancement method for long context processing based on neuronal dynamic adjustment.

[0021] Compared with the prior art, the beneficial effects of the present invention include at least the following: 1. Dynamically Adaptive Perception of Anomaly Accumulation in Long Contexts: This invention adopts an adaptive anomaly detection threshold strategy based on the current batch statistics, which can track the dynamic evolution of activation distribution as the sequence length increases during long context processing in real time, adaptively adjust the anomaly detection threshold, and effectively capture the phenomenon of increased anomaly accumulation in the later part of long sequences.

[0022] 2. Deterministic hard gating for precise isolation of anomalous components: This invention uses a binary hard mask to deterministically isolate activated anomalous components. Compared with continuous gating functions (such as Sigmoid), hard gating can output 0 or 1 when the absolute value of the gating input is small. This not only efficiently filters out anomalous components accumulated over long contexts, but also preserves a large gradient, avoiding the gradient vanishing problem in the saturation region of continuous gating.

[0023] 3. Learnable hierarchical compression protects key context information: This invention introduces independent learnable compression parameters for each feature dimension, enabling the model to adaptively learn the optimal suppression strength of abnormal components at different positions and feature dimensions in long contexts, avoiding damage to normal activation values ​​carrying key semantic information by "one-size-fits-all" compression.

[0024] 4. Broad compatibility and plug-and-play features: As a plug-and-play neural modulation unit, this invention can be seamlessly integrated with existing parameter fine-tuning methods such as LoRA, DoRA, AdaLoRA, and LoRA-Pro without modifying the structure of the method itself. It continuously improves the robustness and performance of long context processing tasks with less than 0.5% additional parameters.

[0025] 5. Significantly improved performance on long-context tasks: On long text reading comprehension tasks, this method improves the model's accuracy by 1.2% with a context length of 32K; on the GSM8K multi-step reasoning task for mathematical word problems, the accuracy is improved by 1.6 percentage points; and on the long document summarization task, the ROUGE-L score is improved by 0.9%. Simultaneously, the loss curve during training is smoother, the effective rank is maintained at a higher level, and the gradient distribution between layers is more uniform. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the overall architecture of the robust enhancement method for long context processing based on dynamic neuron modulation of the present invention. Figure 2 This invention uses an adaptive anomaly detection threshold to automatically capture outliers; Figure 3 This invention performs the process of activating hard-gated truncation; Figure 4 This invention performs a hierarchical response compression process; Figure 5 The present invention demonstrates the stable activation values ​​of the neuron's dynamic regulation output; Figure 6 , Figure 7 ,as well as Figure 8 The training loss curves of the present invention are compared on three efficient parameter fine-tuning methods: LoRA, DoRA, and LoRA-SB. Figure 9 and Figure 10 The figures show a comparison of the effective rank and outlier evolution of the original LoRA and the present invention on long context tasks. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0029] Taking the application to long text reading comprehension tasks as an example, this invention provides a detailed explanation of the robust enhancement method for long context processing based on dynamic neuron regulation.

[0030] When applying the method of this invention to long text reading comprehension tasks, the dataset used is a multi-document question-answering subset from LongBench, with an average document length of 15K tokens and a maximum length of 32K tokens. The base large language model is LLaMA2-7B, and the efficient parameter fine-tuning method is LoRA (rank r=32). Figure 1 The method of the present invention includes the following steps performed sequentially: S1 embeds a neural modulation unit in each Transformer layer of a large language model used for long context processing tasks.

[0031] In this embodiment, a neural modulation unit is embedded in each Transformer layer of LLaMA2-7B. Specifically, the neural modulation unit is inserted after the multi-head self-attention sublayer and before the feedforward network sublayer in each Transformer layer. The input of the neural modulation unit is the output activation tensor of the multi-head self-attention sublayer, and the output is the activation tensor after suppressing anomalies and stabilizing, which is then fed into the subsequent feedforward network sublayer.

[0032] S2, The neural modulation unit performs dynamic activation of abnormal perception: it adaptively calculates the abnormal detection threshold of the current Transformer layer based on the statistical characteristics of the activation value of the current Transformer layer.

[0033] During the forward propagation of each training batch, the activation tensor of the multi-head self-attention sublayer output in the current Transformer layer is... (In this embodiment) =4, =32768, =4096), calculate the feature norm of each token. Then calculate the 90th percentile of the norm vector. And the median, the final anomaly detection threshold is . Figure 2 This invention demonstrates the automatic capture of outliers using an adaptive threshold. Experiments show that when the sequence length exceeds 16K tokens, The value increases significantly, and the adaptive threshold of this invention can automatically capture this change.

[0034] S3, the neural regulation unit performs activation hard gating truncation: a binary hard mask is generated based on the anomaly detection threshold. The binary hard mask achieves deterministic isolation of the abnormal components in the activation values, while keeping the normal components in the normal activation values ​​unchanged.

[0035] For each element in the activation tensor H If its absolute value exceeds the anomaly detection threshold Then the mask =1 otherwise 0. Statistically, in this embodiment, the proportion of mask values ​​of 1 is approximately 0.3% in the first half of the sequence and rises to approximately 0.6% in the second half (after exceeding 16K tokens), indicating that the threshold of this invention adaptively responds to the accumulation of anomalies in a long context. This mask is separated from the computation graph during backpropagation and serves only as a constant coefficient for gradient selection.

[0036] like Figure 3 As shown, this binary hard mask is used to truncate abnormal and normal components.

[0037] S4, the neural modulation unit performs hierarchical response compression, including: performing learnable dimensional linear compression on the isolated abnormal components, reintegrating the compressed abnormal components with the identity-passing normal components through gating fusion, and outputting the activation tensor after suppressing the abnormality.

[0038] like Figure 4 As shown, for abnormal components ( (position = 1), application can learn compression: ,in ∈ The learnable parameters for each feature dimension are initialized to 0 (corresponding to a compression factor of 0.5). For normal components ( (position = 0), execute identity passthrough: = Finally, gating fusion was implemented. Output stable activation values, such as Figure 5 As shown.

[0039] In this embodiment, the neural modulation unit is also jointly optimized with the large language model, specifically by optimizing the learnable compression parameters of the neural modulation unit. Joint optimization with the low-rank matrix parameters of LoRA was performed. The optimizer used was AdamW with a learning rate of 2e-4, trained for 3 epochs. During training, the anomaly detection threshold was dynamically updated with each batch; during inference, the final anomaly detection threshold saved at the end of training was used.

[0040] Experimental Results: On the LongBench multi-document question answering test set, the neural modulation unit of this invention combined with the LoRA method achieved an accuracy of 68.7% (F1 score), which is a 1.2% improvement compared to the original LoRA (67.5%) and a 0.8% improvement compared to full parameter fine-tuning (67.9%). In particular, the improvement of this method is more significant on samples with document lengths exceeding 20K tokens (from 64.2% to 66.1%, an improvement of 1.9%), indicating that this invention has a prominent effect on suppressing anomaly accumulation in long context scenarios.

[0041] To verify the necessity of each core step in the neural modulation unit of this invention, an ablation experiment was conducted on a long text reading comprehension task using LLaMA2-7B combined with LoRA: Ablation experiments show that the three core components—adaptive thresholding, hard gating truncation, and learnable dimensionality compression—all contribute positively to the final performance of this invention, and their synergistic effect achieves the best results.

[0042] Figure 6 , Figure 7 ,as well as Figure 8 The training loss curves of the present invention on three parameter-efficient fine-tuning methods—LoRA, DoRA, and LoRA-SB—are compared. The original LoRA showed a loss as high as 2.9 in the early stages of training, accompanied by severe oscillations. The original DoRA and original LoRA-SB both exhibited significant loss rebounds in the 50-150 step range (rebounding from 0.4 to 0.8 and from 0.5 to 0.8, respectively). After enhancement with the neural modulation unit of the present invention, although the initial losses of the three methods were relatively high (approximately 4.9-5.0), they rapidly decreased within the first 10 steps. The loss curves became smoother during subsequent training, effectively suppressing oscillations and rebounds, and significantly improving training stability.

[0043] Figure 9 and Figure 10 The evolution of the effective rank during training is compared between the original LoRA and the present invention. The effective rank of the original LoRA continuously decreased during training, from approximately 193 (50 steps) to approximately 100 (200 steps), then further to approximately 45 (12800 steps), and finally maintained at a low level of approximately 45-50 in the later stages of training, indicating significant rank collapse. However, after incorporating the neural modulation unit of the present invention with LoRA, the effective rank initially decreased from approximately 195 (50 steps) to approximately 135 (800 steps), then rebounded and stabilized in the range of approximately 150-170 (12800 to 51200 steps), demonstrating that the present invention effectively suppresses rank collapse caused by long sequences and suppresses outliers, protecting the representational health of the model.

[0044] The embodiments also apply the invention to multi-step reasoning tasks involving mathematical word problems. The dataset uses GSM8K and contains approximately 8,000 elementary school math word problems, each requiring 2-8 steps of reasoning to arrive at the correct answer. The base models used are Mistral-7B and Gemma-2 9B, and efficient parameter fine-tuning methods include LoRA, DoRA, LoRA-Pro, and LoRA-SB.

[0045] Experimental Results: For the Mistral-7B model, the LoRA-Pro method enhanced with the neural modulation unit of this invention achieves an accuracy of 63.7% on GSM8K, a 0.6% improvement over the original LoRA-Pro (63.1%). The DoRA method enhanced with the neural modulation unit achieves 17.3% accuracy on the MATH (Mathematics Aptitude Test of Heuristics, proposed by Hendrycks et al. in 2021, containing 12,500 challenging mathematical competition problems covering algebra, geometry, number theory, probability, and other fields) benchmark, a 0.7% improvement over the original DoRA (16.6%). For the Gemma-2 9B model, the DoRA method enhanced with the neural modulation unit achieves 79.2% accuracy on the GSM8K dataset, a 1.6% improvement over the original DoRA (77.6%); and 37.8% accuracy on the MATH benchmark, a 0.8% improvement over the original DoRA (37.0%).

[0046] Of particular note is that the improvement of this invention is more significant on problems requiring longer reasoning chains (more than 5 steps) (from 55.3% to 57.8% on the GSM8K dataset, an improvement of 2.5%), indicating that this invention effectively protects the intermediate step information in the long chain reasoning process and prevents reasoning interruption or errors caused by the accumulation of activation anomalies.

[0047] This embodiment applies the method of the present invention to a long document summarization task. The dataset used is the PubMed dataset, which contains summaries of approximately 200,000 biomedical papers as the target for generation, with an average source document length of 3000 tokens. The base model is LLaMA2-7B, and the LoRA method is used for efficient parameter fine-tuning.

[0048] Experimental Results: On the ROUGE-L evaluation metric, the neural modulation unit of this invention combined with the LoRA method achieved 48.2%, an improvement of 0.9% compared to the original LoRA (47.3%). Manual evaluation showed that the summary generated by this method more completely preserved the key information of the original text, especially the information retention rate of the last third of the document, which increased from 78% to 84%, indicating that this invention effectively suppressed the attenuation of information in the latter part of long documents during processing.

[0049] The embodiment also provides a robust enhancement system for long context processing based on dynamic neuron modulation, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned robust enhancement method for long context processing based on dynamic neuron modulation, specifically including the following steps: S1, embed a neural modulation unit in each Transformer layer of a large language model used for long context processing tasks; S2, The neural modulation unit performs dynamic activation anomaly perception: adaptively calculates the anomaly detection threshold of the current Transformer layer based on the statistical characteristics of the activation value of the current Transformer layer; S3, the neural modulation unit performs activation hard gating cutoff: a binary hard mask is generated based on the anomaly detection threshold. The binary hard mask achieves deterministic isolation of the abnormal components in the activation values, while keeping the normal components in the normal activation values ​​unchanged. S4, the neural modulation unit performs hierarchical response compression, including: performing learnable dimensional linear compression on the isolated abnormal components, reintegrating the compressed abnormal components with the identity-passing normal components through gating fusion, and outputting the activation tensor after suppressing the abnormality.

[0050] The robust enhancement method for long context processing based on dynamic neuron regulation provided in this embodiment, at the hardware level, includes not only a processor and memory, but also other hardware required for the service, such as an internal bus, network interface, and memory. The memory is non-volatile; the processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the robust enhancement method for long context processing based on dynamic neuron regulation described in S1-S4 above. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0051] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described robust enhancement method for long context processing based on dynamic neuron modulation, specifically including the following steps: S1, embed a neural modulation unit in each Transformer layer of a large language model used for long context processing tasks; S2, The neural modulation unit performs dynamic activation anomaly perception: adaptively calculates the anomaly detection threshold of the current Transformer layer based on the statistical characteristics of the activation value of the current Transformer layer; S3, the neural modulation unit performs activation hard gating cutoff: a binary hard mask is generated based on the anomaly detection threshold. The binary hard mask achieves deterministic isolation of the abnormal components in the activation values, while keeping the normal components in the normal activation values ​​unchanged. S4, the neural modulation unit performs hierarchical response compression, including: performing learnable dimensional linear compression on the isolated abnormal components, reintegrating the compressed abnormal components with the identity-passing normal components through gating fusion, and outputting the activation tensor after suppressing the abnormality.

[0052] In this embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.

[0053] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robust enhancement method for long context processing based on dynamic neuron modulation, characterized in that, Includes the following steps: A neural modulation unit is embedded in each Transformer layer of the large language model used for long context processing tasks. The neural modulation unit sequentially performs dynamic activation of anomaly perception, activation of hard gating truncation, and hierarchical response compression steps. The dynamic activation anomaly perception includes: adaptively calculating the anomaly detection threshold of the current Transformer layer based on the statistical characteristics of the activation value of the current Transformer layer; The activation hard gating cutoff includes: generating a binary hard mask based on an anomaly detection threshold. The binary hard mask enables deterministic isolation of the abnormal components in the activation values, while keeping the normal components in the normal activation values ​​unchanged. The hierarchical response compression includes: performing learnable dimensional linear compression on the isolated anomalous components, reintegrating the compressed anomalous components with the identity-passing normal components through gated fusion, and outputting the activation tensor after suppressing the anomalous components.

2. The robust enhancement method for long context processing based on neuronal dynamic regulation according to claim 1, characterized in that, The adaptive calculation of the anomaly detection threshold for the current Transformer layer based on the statistical characteristics of the activation values ​​of the current Transformer layer includes: For the activation tensor of the current Transformer layer ,in For batch size, For sequence length, For feature dimension, If it is a tensor space, then the anomaly detection threshold is... Calculated using the following dual-benchmark strategy: in, The norm of the current batch activation value The 90th percentile was used as an adaptive baseline; The median of the norm of the current batch of activation values. 1.0 serves as an absolute benchmark to prevent the under-detection of extremely small outliers; 1.0 is the lower limit of numerical stability. To find the maximum value.

3. The robust enhancement method for long context processing based on neuronal dynamic regulation according to claim 1, characterized in that, Generate a binary hard mask based on the anomaly detection threshold, including: For activation tensor H Each element in Generate the corresponding binary mask. : in This is an indicator function that determines when the activation value norm exceeds a threshold. The time mask value is 1 if it is not 0 otherwise. subscript These are batch index, sequence long position index, and feature dimension index, respectively.

4. The robust enhancement method for long context processing based on neuronal dynamic modulation according to claim 1, characterized in that, Learnable dimensional linear compression of the isolated anomalous components, including: For abnormal components that are marked as abnormal by the mask Apply the following compression transformation: in, , For feature dimension, Compression parameters that are learnable independently for each feature dimension. The sigmoid function constrains the compression coefficient to the interval (0,1). This indicates element-wise multiplication. These are the compressed anomalous components; For normal components that are masked and marked as normal Perform constant pass-through, that is, constant pass-through of the normal component. .

5. The robust enhancement method for long context processing based on neuronal dynamic regulation according to claim 1, characterized in that, Gated fusion is used to reintegrate compressed anomalous components with normal components that pass through the identity directly, including: Where 1 represents a tensor of all 1s. The activation value output by the neural regulatory unit after the inhibition of abnormality. The original activation value is replaced and fed into subsequent sub-layers of the current Transformer layer. These are the compressed anomalous components. This is the normal component that passes through directly with equal identity. This indicates element-wise multiplication. It is a binary mask.

6. The robust enhancement method for long context processing based on neuronal dynamic modulation according to claim 1, characterized in that, The neural modulation unit dynamically updates the anomaly detection threshold in training mode and uses the final anomaly detection threshold saved at the end of training in inference mode. Furthermore, the learnable compression parameters in hierarchical response compression are jointly optimized with other parameters of the large language model during the model training process for long context processing tasks.

7. The robust enhancement method for long context processing based on neuronal dynamic modulation according to claim 1, characterized in that, The long context processing tasks include at least one of the following: long text reading comprehension, multi-step reasoning for mathematical word problems, long document summarization generation, multi-turn dialogue context understanding, and cross-line dependency processing in code generation.

8. A robust enhancement system for long context processing based on dynamic neuron modulation, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the one or more processors execute the executable code, they are used to implement the robust enhancement method for long context processing based on neuronal dynamic adjustment as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the robust enhancement method for long context processing based on neuronal dynamic modulation as described in any one of claims 1-7.

Images