Method and system for improving memory capacity of large model

Abstract

The invention relates to the technical field of large models, in particular to a method and system for improving the memory amount of a large model, and the method comprises the steps: deploying a monitoring module in a training or reasoning process, carrying out the statistics of the number of memory errors and the total task amount of each component, and calculating the initial memory performance of the components; based on the performance evaluation result of each component, quantifying the overall initial memory amount of the model as an optimization starting point; utilizing a dynamic sensitivity analysis algorithm to mine the influence degree of each component parameter on the memory performance, and calculating a dynamic sensitivity factor; and in combination with the sensitivity factor, the current parameter state and the memory performance, model parameters are adaptively adjusted through dynamic parameter optimization and an iterative lifting algorithm, an iterative verification mechanism is introduced, and the memory ability of each component and the whole machine is subjected to closed-loop optimization. According to the method, fine quantization and parameter-level dynamic regulation and control on the memory performance of the model are realized, and the memory stability and long-term dependence processing capability of the large model in tasks such as knowledge keeping and context understanding are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and in particular to a method and system for improving the memory capacity of large models. Background Technology

[0002] With the rapid development of artificial intelligence technology, large models are increasingly widely used in fields such as natural language processing, computer vision, and speech recognition. The ever-increasing model size brings stronger expressive and reasoning capabilities, but it also places higher demands on the model's memory capacity. The memory capacity of a large model refers to its ability to effectively store and accurately recall historical information, knowledge parameters, and contextual relationships during training and inference. However, in practical applications, due to issues such as parameter redundancy, gradient sparsity, attention mechanism biases, and training data noise, large models often exhibit forgetting phenomena or memory errors, such as contextual understanding biases, inaccurate knowledge recall, and breaks in long sequence dependencies, severely affecting the model's stability and generalization performance.

[0003] Existing methods for improving model memory capabilities mostly focus on structural adjustments (such as introducing external memory units), optimizing training strategies (such as continuous learning and replay mechanisms), or increasing model capacity. However, these methods often lead to problems such as a surge in computational overhead, training instability, or diminishing marginal returns. Meanwhile, existing parameter tuning processes largely rely on empirical parameter tuning or global optimization algorithms, lacking refined modeling of the differences in memory performance among different components (such as attention heads, feedforward networks, and layer normalization modules), and failing to fully consider the dynamic impact of various model parameters on memory function and their inter-coupling relationships.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a method and system for improving the memory capacity of large models. This aims to solve the technical problem that existing large models lack quantitative analysis and parameter-level dynamic optimization mechanisms for the memory performance of each component, which leads to limitations in memory capacity improvement based on empirical parameter tuning and overall structural modifications, making it difficult to achieve efficient and accurate memory capacity enhancement.

[0006] To achieve the above objectives, the present invention provides a method for increasing the memory capacity of large models, the method comprising:

[0007] Deploy a monitoring module during model training or inference to record the number of memory errors and the total number of tasks processed for each component of the large model during the task processing phase, and calculate the initial memory performance of each component based on the number of memory errors and the total number of tasks processed.

[0008] Based on the initial memory performance of each component, the initial memory size of the entire large model is calculated as the starting point for dynamic parameter optimization.

[0009] The relationship between component memory performance and model parameters is analyzed using a dynamic sensitivity analysis algorithm, and the dynamic sensitivity factor of each component model parameter is calculated.

[0010] Based on dynamic sensitivity factors, component memory performance, and model parameters, the model parameters are dynamically adjusted through dynamic parameter optimization and memory capacity iteration improvement algorithms. Combined with sensitivity analysis and iterative verification, the memory capacity of components and the whole machine is gradually increased.

[0011] Optionally, the dynamic sensitivity analysis algorithm evaluates the direct impact of changes in model parameters on the component's memory performance by taking the derivative of the experimentally fitted curve with controlled variables to obtain the partial derivative of the component's memory performance with respect to the model parameters; the partial derivative is multiplied by the ratio of the component's model parameter value to the component's memory performance.

[0012] Introducing a nonlinear influence factor, we describe the impact of model parameters deviating from the reference value using an exponential decay function, simulating the nonlinear effect on component memory performance when model parameters deviate from the optimal state during actual training.

[0013] Considering the interaction between model parameters, the correlation between each model parameter and other model parameters is calculated using statistical methods. The covariance and variance between model parameters are calculated to obtain a standardized interaction weight.

[0014] Optionally, the formula for calculating the dynamic sensitivity factor is:

[0015]

[0016] in, This represents the dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration; This represents the memory performance of the i-th component in the t-th iteration, initially... , This represents the initial memory performance of the i-th component; This represents the j-th model parameter value of the i-th component in the t-th iteration; This represents the partial derivative of the component's memory performance with respect to the model parameter values; This represents the relative proportion of model parameters to component memory performance; Indicates the nonlinear influence factor; This represents the reference value of the j-th model parameter of the i-th component; Indicates the attenuation coefficient; This represents the weighted sum of the interaction effects of model parameters; This represents the interaction weight between the j-th and k-th model parameters.

[0017] Optionally, the dynamic parameter optimization and memory capacity iteration improvement algorithm calculates the model parameter adjustment amount based on the dynamic sensitivity factor and component memory performance, and reasonably determines the adjustment range of the model parameters;

[0018] By calculating the unmemorable degree of the current component's memory performance, multiplying the unmemorable degree by the dynamic sensitivity factor, a preliminary adjustment range is obtained as the numerator. The sum of the square of the dynamic sensitivity factor and a very small positive number is introduced into the denominator. At the same time, an adjustment coefficient is introduced to allow control of the adjustment step size according to actual needs.

[0019] Optionally, the formula for calculating the model parameter adjustment amount is as follows:

[0020]

[0021] in, This represents the adjustment amount of the j-th model parameter of the i-th component in the t-th iteration; Indicates the adjustment coefficient; The memorability of the i-th component in the t-th iteration is represented by its unmemorability. This represents the memory performance of the i-th component in the t-th iteration; Indicates the normalized denominator; Represents a very small positive number; This represents the dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration.

[0022] Optionally, the dynamic parameter optimization and memory iteration improvement algorithm updates the model parameter values ​​based on the current model parameter values ​​and the model parameter adjustment amount. To ensure that the adjusted model parameters are computationally feasible, boundary constraints are introduced. The sum of the current model parameter values ​​and the model parameter adjustment amount is compared with the maximum allowable value of the model parameters, and the smaller value is taken. Then, it is compared with the minimum allowable value of the model parameters, and the larger value is taken as the final updated model parameter value, ensuring that the adjusted parameters do not exceed the limitations of hardware resources or model architecture.

[0023] The memory performance of the component is further updated based on the updated model parameter values;

[0024] The adjusted model parameters directly reflect the changes in component memory performance. The actual change in model parameters in this iteration is calculated by subtracting the model parameter value before adjustment from the updated model parameter value and dividing by the reference value of the model parameter to obtain a dimensionless relative change magnitude, which is then multiplied by the corresponding dynamic sensitivity factor. The non-memory factor is introduced to make the improvement in component memory performance gradually decrease as it approaches the theoretical upper limit.

[0025] Optionally, the dynamic parameter optimization and memory capacity iterative improvement algorithm calculates the memory capacity of the entire large model based on the memory performance of the components after iteration, and evaluates the error between the memory capacity of the entire large model and the target value. If the error is positive, it means that the current memory capacity of the entire large model has not yet reached the target, that is, the effect of the current model parameter adjustment is insufficient, and iterative improvement is needed. If the error is zero or negative, it means that the target has been achieved or exceeded, and the iteration process is stopped.

[0026] Furthermore, to achieve the above objectives, the present invention also provides a system for improving the memory capacity of large models, the system comprising:

[0027] The memory monitoring module is used to deploy a monitoring module during model training or inference to record the number of memory errors and the total number of tasks processed for each component of the large model during the task processing phase, and to calculate the initial memory performance of each component based on the number of memory errors and the total number of tasks processed.

[0028] The performance evaluation module is used to calculate the initial memory size of the entire large model based on the initial memory performance of each component, serving as the starting point for dynamic parameter optimization.

[0029] The sensitivity analysis module is used to analyze the relationship between component memory performance and model parameters through dynamic sensitivity analysis algorithms, and to calculate the dynamic sensitivity factor of each component model parameter;

[0030] The parameter optimization module is used to dynamically adjust model parameters based on dynamic sensitivity factors, component memory performance, and model parameters through dynamic parameter optimization and memory capacity iteration improvement algorithms. Combined with sensitivity analysis and iterative verification, it gradually increases the memory capacity of components and the whole machine.

[0031] Furthermore, to achieve the above objectives, the present invention also provides an apparatus for increasing the memory capacity of large models, the apparatus comprising: a memory, a processor, and a program for increasing the memory capacity of large models stored in the memory and executable on the processor, the program for increasing the memory capacity of large models being configured to implement the steps of the method for increasing the memory capacity of large models as described above.

[0032] In addition, to achieve the above objectives, the present invention also provides a medium storing a program for increasing the memory capacity of large models, wherein when the program for increasing the memory capacity of large models is executed by a processor, the program for increasing the memory capacity of large models implements the steps of the method for increasing the memory capacity of large models as described above.

[0033] This invention provides a method for improving the memory capacity of large models. The method achieves refined quantitative evaluation of the memory performance of each component of the large model by deploying a monitoring module, accurately identifies the degree of influence of key parameters on memory capacity by combining dynamic sensitivity analysis algorithm, and then adaptively adjusts model parameters using dynamic parameter optimization and iterative improvement mechanism. Without increasing the model size, the memory capacity of components and the overall model is effectively improved. This method achieves interpretability, dynamism and closed-loop optimization of memory enhancement, significantly improves the stability and generalization ability of large models in long sequence modeling, knowledge preservation and context understanding, while avoiding the high cost and low efficiency problems caused by traditional methods that rely on empirical parameter tuning or structural modification. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating an embodiment of the method for increasing the memory capacity of large models according to the present invention.

[0035] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0036] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0037] Reference Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the method for improving the memory capacity of large models according to the present invention, and an embodiment of the method for improving the memory capacity of large models according to the present invention is presented.

[0038] In one embodiment, the method for increasing the memory capacity of large models includes:

[0039] Step S100: Deploy a monitoring module during model training or inference to record the number of memory errors and the total number of tasks processed for each component of the large model during the task processing phase, and calculate the initial memory performance of each component based on the number of memory errors and the total number of tasks processed.

[0040] The monitoring module can be a software or logic unit embedded during model training or inference to collect and record component-level memory behavior data. It can be used to perform fine-grained statistics on the number of memory errors and the total number of tasks processed by each component of the large model in a real task flow, providing a data foundation for initial memory performance evaluation. In this embodiment, the monitoring module can capture the input, output, and task results of each component in real time by inserting hooks or callback functions into the model's forward or backward propagation path, and compare the expected and actual responses to identify memory errors. For example, the monitoring module can include, but is not limited to, one or more of the following: attention head-based monitors, feedforward network layer-based monitors, and layer normalization module-based monitors.

[0041] It should be understood that this memory system employs two processing methods: short-term memory and long-term memory. For short-term memory, a text model is used to extract entities, intentions, and constraints from the dialogue, compressing the memory volume. For long-term memory, recent information is prioritized during retrieval, and related entities are linked through a knowledge graph to avoid confusion due to names being the same. The entity recognition model for short-term memory requires the extraction of entities, intentions, and constraints. Transfer training is performed using a BERT-base / Chinese basic model. By organizing training data of Chinese entities, intentions, and constraints, and then performing transfer training, the resulting model has low memory usage and is significantly faster than larger models. The extracted data is sent to the larger model along with the dialogue history and new questions. This data volume is far smaller than simply concatenating dialogue history data. For long-term memory, the time-series database Prometheus is used to store dialogue history data. Leveraging the characteristics of time-series databases, the weight of recent information is increased, allowing for the priority return of recent information, as recent information content often receives higher attention during the dialogue. Based on the BERT-Base / Large basic text model, we organized the Chinese entity association training set and trained an entity association model that can extract entity associations. By extracting association relationships, the amount of data in dialogue memory is greatly reduced.

[0042] Each component of a large model can be a basic functional unit that constitutes the large model, such as an attention head, a feedforward network, or a layer normalization module. These components can undertake specific computational tasks and participate in information storage and retrieval during context processing; their performance directly affects the overall memory capacity. Furthermore, each component of a large model can include, but is not limited to, a single attention head in a multi-head attention mechanism, a feedforward neural network in a Transformer block, or a layer normalization unit in a residual connection.

[0043] The number of memory errors can be the number of times a component fails to correctly retrieve or maintain historical information, knowledge parameters, or contextual associations when processing tasks. It can be used as a negative indicator of component memory reliability to calculate initial memory performance. In an exemplary embodiment, the number of memory errors can be determined by the monitoring module by comparing the model output with the standard answer or consistency constraints. The total number of tasks processed can be the total number of task instances processed by the component within a specific time period or data batch. It can be used as a normalization benchmark for the number of memory errors, ensuring the fairness and comparability of memory performance evaluation. Initial memory performance can be a component-level memory capability quantification indicator calculated based on the number of memory errors and the total number of tasks processed. It can be used to reflect the component's memory stability under the current parameter configuration, serving as a starting point for subsequent sensitivity analysis and parameter optimization. In a specific embodiment, initial memory performance can be calculated using forms such as accuracy, the reciprocal of the error rate, or a weighted success rate.

[0044] Deploying a monitoring module during model training or inference can be achieved by embedding monitoring logic into the model execution flow, synchronously collecting behavioral data as data flows through each component. Furthermore, deploying a monitoring module during model training or inference can be accomplished by integrating the monitoring logic into the model computation graph, thereby establishing observability of component-level memory behavior and providing raw data support for subsequent quantitative evaluation.

[0045] Recording the number of memory errors and the total number of tasks processed for each component of the large model during the task processing phase can be done for each task instance. The monitoring module compares the component output with the expected memory behavior, accumulating errors and the total number of processing times. Furthermore, recording the number of memory errors and the total number of tasks processed for each component of the large model during the task processing phase can be achieved by synchronously counting in the validation task flow, thereby generating standardized statistics that can be used for performance evaluation and supporting fine-grained memory capability diagnosis.

[0046] The initial memory performance of each component is calculated based on the number of memory errors and the total number of tasks processed. This can be achieved by converting raw statistics into a performance score using ratios, success rates, or weighted metric formulas. Furthermore, calculating the initial memory performance of each component based on the number of memory errors and the total number of tasks processed can be accomplished through online sliding window statistics or batch aggregation, thereby transforming discrete error events into continuously comparable memory performance metrics.

[0047] Step S200: Based on the initial memory performance of each component, calculate the initial memory size of the entire large model as the starting point for dynamic parameter optimization.

[0048] The initial memory capacity of the entire large model can be a global memory capacity metric aggregated from the initial memory performance of all components. This metric can be used to characterize the overall memory level of the model and provide an initial reference value for dynamic parameter optimization. In an exemplary embodiment, the initial memory capacity of the entire large model can be a combination of the performance of each component through weighted averaging, max pooling, or a fusion strategy based on component importance.

[0049] Integrating the initial memory performance of all components by pre-defined aggregation rules can be achieved through methods such as weighted averaging. Furthermore, integrating the initial memory performance of all components by pre-defined aggregation rules can be accomplished by assigning higher weights to frequently used components, thereby establishing a global memory capacity baseline for measuring subsequent optimization progress.

[0050] Step S300: Analyze the relationship between component memory performance and model parameters using a dynamic sensitivity analysis algorithm, and calculate the dynamic sensitivity factor of each component model parameter.

[0051] Dynamic sensitivity analysis algorithms can be adaptive analysis methods used to quantify the impact of model parameter variations on component memory performance. They can accurately identify parameters that play a key role in memory capability and the strength of their dynamic influence, overcoming the limitations of static or global sensitivity analysis. In one specific embodiment, dynamic sensitivity analysis algorithms can introduce local parameter perturbations during training or inference, observe changes in memory performance, and fit the parameter-performance mapping relationship by combining gradient information or a surrogate model. For example, dynamic sensitivity analysis algorithms may include, but are not limited to, gradient-based local sensitivity analysis, surrogate model-based black-box sensitivity estimation, and task replay-based perturbation response analysis.

[0052] Model parameters can be learnable numerical variables in a large model, including weight matrices, bias vectors, normalization parameters, etc. They determine the model's mapping ability and internal representation structure, and their configuration directly affects the storage and retrieval efficiency of memory information. Dynamic sensitivity factors can be time-varying numerical indicators characterizing the degree of influence of specific model parameters on component memory performance. They can be used to identify which parameters have a high influence on memory function, guiding the priority and direction of subsequent parameter adjustments. In an exemplary embodiment, the dynamic sensitivity factor can be calculated through the response relationship between parameter perturbations and changes in memory performance. Exemplarily, dynamic sensitivity factors may include, but are not limited to, attention weight sensitivity factors, feedforward network bias sensitivity factors, and layer normalization scaling factor sensitivity factors.

[0053] Analyzing the relationship between component memory performance and model parameters using dynamic sensitivity analysis algorithms can be achieved by applying small perturbations to the parameter space, observing changes in the corresponding component memory performance, and establishing a mapping model. Furthermore, this analysis can be implemented through local linear approximation or nonparametric regression, revealing the causal impact of parameters on memory function and providing a basis for targeted optimization.

[0054] The dynamic sensitivity factor for each component model parameter can be calculated by determining the sensitivity value based on the ratio of parameter perturbation magnitude to memory performance change or the fitting coefficient. Furthermore, the dynamic sensitivity factor for each component model parameter can be calculated using sliding window historical performance differencing or surrogate model gradient estimation, thereby quantifying the contribution of each parameter to memory capability and distinguishing between critical and redundant parameters.

[0055] Step S400: Based on the dynamic sensitivity factor, component memory performance and model parameters, the model parameters are dynamically adjusted through dynamic parameter optimization and memory capacity iteration improvement algorithms, and combined with sensitivity analysis and iterative verification, the memory capacity of the components and the whole machine is gradually increased.

[0056] The dynamic parameter optimization and memory capacity iteration improvement algorithm can be a closed-loop optimization mechanism that combines sensitivity factors and memory performance feedback to adaptively adjust model parameters to improve memory capacity. It can be used to achieve refined and adaptive adjustment of key memory-related parameters without increasing model size, continuously improving the memory capabilities of components and the entire system. In a specific embodiment, the dynamic parameter optimization and memory capacity iteration improvement algorithm can apply targeted updates to high-impact parameters based on a dynamic sensitivity factor in each iteration, and evaluate the memory capacity improvement effect through a verification task, forming a feedback loop. For example, the dynamic parameter optimization and memory capacity iteration improvement algorithm may include, but is not limited to, sensitivity-weighted gradient descent optimization, memory performance-driven parameter reinitialization strategies, and multi-component collaborative memory-oriented fine-tuning algorithms.

[0057] Applying larger update step sizes or specific optimization strategies to high-influence parameters based on sensitivity factors, while retaining low-sensitivity parameters, can be achieved by weighting the gradient according to the sensitivity factor during backpropagation before updating the parameters. Furthermore, applying larger update step sizes or specific optimization strategies to high-influence parameters based on sensitivity factors while retaining low-sensitivity parameters can be achieved by training only a subset of high-sensitivity parameters during the fine-tuning phase. This allows for differentiated control of parameter updates and improves the optimization efficiency of memorizing relevant parameters.

[0058] After each round of parameter adjustments, the monitoring module is rerun to evaluate memory performance, forming an analysis-optimization-validation closed loop. This can be achieved by running long-sequence tasks on the validation set to test context preservation capabilities. Furthermore, this analysis-optimization-validation closed loop, where the monitoring module is rerun after each round of parameter adjustments to verify the accuracy of historical information retrieval, can be implemented through knowledge backtracking tasks. This ensures the stability and generalization of the memory capacity improvement, avoiding overfitting or performance fluctuations.

[0059] Taking a long document question answering system as an example, the method to improve the memory capacity of a large model in this embodiment can be as follows: when processing a question answering task of a technical manual containing multiple chapters, the monitoring module records the number of errors made by each attention head of the Transformer when answering cross-chapter questions; dynamic sensitivity analysis reveals that the bias parameters of certain feedforward network layers are highly sensitive to the resolution of inter-chapter references; the dynamic parameter optimization algorithm accordingly fine-tunes these parameters, and verifies the improvement in the model's accuracy in answering questions such as "whether the method mentioned in Chapter 3 is applicable to the scenario in Chapter 5" in subsequent iterations, thereby enhancing the ability to model long-distance dependencies without increasing the number of model layers or the total number of parameters.

[0060] This embodiment deploys a monitoring module to perform fine-grained statistical analysis of memory errors in each component of a large model during real-world task flows, achieving a quantitative evaluation of component-level memory performance. This overcomes the limitations of traditional methods that rely solely on overall metrics or structural modifications. Building upon this, a dynamic sensitivity analysis algorithm is introduced to accurately characterize the dynamic impact of different model parameters on component memory capabilities, addressing the issue of existing parameter tuning strategies neglecting the parameter-memory coupling relationship. Furthermore, by utilizing dynamic parameter optimization and iterative memory enhancement algorithms, with sensitivity factors as the optimization guide, adaptive, closed-loop adjustments to key memory-related parameters are achieved without expanding the model size. This not only improves the memory stability of individual components but also enhances the overall system's generalization capabilities in long-sequence dependency modeling, context consistency maintenance, and knowledge backtracking through inter-component collaborative optimization. Simultaneously, it avoids the blindness of empirical parameter tuning and the computational overhead of external memory structures, thus achieving a highly efficient, interpretable, and dynamically evolving memory enhancement effect.

[0061] In one embodiment, the dynamic sensitivity analysis algorithm evaluates the direct impact of changes in model parameters on the component's memory performance by taking the derivative of the fitted curve with controlled variables to obtain the partial derivative of the component's memory performance with respect to the model parameters; the partial derivative is then multiplied by the ratio of the component's model parameter value to the component's memory performance.

[0062] One method to assess the direct impact of model parameter changes on component memory performance is to keep other parameters constant and only perturb the target parameter, observing the change in component memory performance. Further, this operation can be achieved by sampling multiple points in the parameter's neighborhood, fitting a local response curve of the memory performance with respect to that parameter, and then calculating its analytical or numerical derivative. This allows us to obtain the parameter's local sensitivity to memory performance, serving as an indicator of the intensity of the impact under a linear approximation. For example, fitting the local response curve can be done by using polynomial or Gaussian process regression to fit the local response surface and then calculating the derivative, or by directly calculating the performance difference quotient under forward and backward perturbations using the finite difference method.

[0063] Multiplying the partial derivative by the ratio of the component model parameter value to the component memory performance can be done by calculating the product of the partial derivative with the current value of that parameter and the current ratio of its memory performance for each parameter. In a specific embodiment, this operation is achieved by calculating (∂M / ∂θ)×(θ / M), where M is the component memory performance and θ is the parameter value, thereby achieving relative normalization of sensitivity and eliminating evaluation bias caused by differences in parameter magnitude or absolute performance values.

[0064] Introducing a nonlinear influence factor, we describe the impact of model parameters deviating from the reference value using an exponential decay function, simulating the nonlinear effect on component memory performance when model parameters deviate from the optimal state during actual training.

[0065] The nonlinear influence factor can be an adjustment coefficient used to characterize the nonlinear decay effect on component memory performance when model parameters deviate from the reference value. It can be used to introduce a penalty mechanism for parameter deviation in sensitivity calculations, allowing parameters far from the optimal region to obtain lower effective sensitivity and more realistically reflect their actual impact on memory performance. In this embodiment, the nonlinear influence factor can be constructed using the exponential decay function exp(-α·|θ-θ0|), where θ is the current parameter value, θ0 is the reference value, and α is the decay rate control parameter. Furthermore, the nonlinear influence factor can include, but is not limited to, one or more of the following: a decay factor based on L1 distance, a decay factor based on L2 distance, and an adaptive decay factor based on task loss curvature.

[0066] An exponential decay function can be a non-linear function that monotonically decreases as the independent variable increases. It is often used to model the effect of rapid decay as the degree of deviation increases. It can be used to quantify the inhibitory effect of parameter deviation from the reference value on memory performance, reflecting the non-linear degradation of memory ability caused by parameter inaccuracies during training. For example, exponential decay functions can take the form of one-sided exponential decay, two-sided symmetrical exponential decay, or piecewise exponential decay.

[0067] Reference values ​​can be benchmarks used to measure whether the current state of model parameters is within the effective memory region. They typically correspond to historical best or initial configurations and can be used as anchor points for calculating nonlinear influence factors, determining the starting position of parameter deviations. In an exemplary embodiment, reference values ​​may include parameter snapshots from the early stages of training, parameter values ​​corresponding to peak memory performance on the validation set, or the mean of parameters within a sliding window.

[0068] Model parameter deviation from the optimal state can be defined as the difference between the current model parameters and the ideal parameter configuration that maximizes component memory performance. This deviation can be used to trigger nonlinear effects and identify parameter configuration regions that may lead to a significant decrease in memory capacity. Nonlinear effects can be the non-proportional, non-constant rate of influence of model parameter changes on component memory performance. They can explain why small parameter perturbations have a significant impact in some regions but a weak impact in others, improving the realism of sensitivity analysis. Introducing a nonlinear influence factor, using an exponential decay function to describe the impact of model parameter deviation from the reference value, can be achieved by using the L1 norm to measure the degree of deviation to enhance sparsity, or by using the L2 norm to smooth the decay curve. This allows sensitivity to automatically decay as parameters deviate from the optimal region, better reflecting the non-convex performance surface characteristics in actual training. Simulating the nonlinear effects of model parameters deviating from the optimal state on component memory performance in actual training can be achieved by embedding a nonlinear decay term in the sensitivity calculation. This ensures that parameters far from the reference value, even with large partial derivatives, obtain lower effective sensitivity, avoiding over-reliance on local linear approximations in suboptimal regions and improving optimization stability.

[0069] Considering the interaction between model parameters, the correlation between each model parameter and other model parameters is calculated using statistical methods. The covariance and variance between model parameters are calculated to obtain a standardized interaction weight.

[0070] The interaction between model parameters can be the coupling or dependency relationship between different model parameters when they jointly affect the memory performance of a component. This can be used to reveal synergistic or offsetting effects that might be overlooked by adjusting isolated parameters, providing a basis for joint optimization. Statistical methods can be a set of mathematical tools used to infer relationships between parameters from observed data. They can support the calculation of indicators such as correlation and covariance, providing a data foundation for constructing interaction weights. In a specific embodiment, statistical methods may include Pearson correlation coefficient estimation, Spearman rank correlation analysis, and local covariance estimation based on a sliding window. Correlation can be the degree of statistical association exhibited by two model parameters in affecting the memory performance of a component, and can be used to initially identify parameter pairs that may have interactive effects.

[0071] Covariance can be a second-order statistic measuring the joint trend of change of two model parameters along the gradient direction of memory performance. It can be used to quantify the joint influence of parameter covariance on memory performance. Variance can be a measure of the degree of fluctuation of a single model parameter in the memory performance response. It can be used to standardize covariance and eliminate the influence of dimensions. Standardized interaction weights can be dimensionless parameter interaction strength indices obtained by dividing the covariance by their respective standard deviations (i.e., correlation coefficients) or other normalization methods. They can be used to adjust the magnitude of joint updates of multiple parameters in dynamic parameter optimization, avoiding optimization oscillations or convergence failures due to ignoring coupling relationships. In this embodiment, standardized interaction weights can be formed by calculating the covariance matrix of the parameter pairs and normalizing it according to the variance to form a symmetric interaction weight matrix. For example, standardized interaction weights can include Pearson correlation weights, partial correlation interaction weights, mutual information weights based on information entropy, etc.

[0072] Considering the interaction between model parameters, the correlation between each model parameter and other model parameters can be calculated using statistical methods. This can be achieved by collecting joint change data of parameters and memory performance in multiple perturbation experiments or training trajectories, calculating the statistical correlation between pairs of parameters, thereby identifying parameter pairs with strong coupling relationships and providing input for interactive modeling.

[0073] The covariance and variance between model parameters can be calculated based on parameter gradient sequences or perturbation response sequences. The covariance matrix and its diagonal elements (variance) can be estimated, thereby quantifying the joint variation pattern of parameters and supporting the standardized calculation of interaction weights.

[0074] A standardized interaction weight can be obtained by dividing the elements of the covariance matrix by the product of the standard deviations of the corresponding parameters, generating a correlation coefficient matrix as the interaction weight. Furthermore, this operation can be achieved by dynamically updating the interaction weights using a sliding time window to adapt to changes during the training phase, or by applying a sparsity threshold to the interaction weights to reduce computational overhead. This provides a structured representation of the collaborative relationships between parameters for dynamic parameter optimization, supporting multi-parameter joint adjustment strategies.

[0075] For example, in the scenario of maintaining referential consistency in long-context dialogue systems, the method for improving the memory capacity of a large model in this embodiment can be as follows: When handling questions like "How is the project he mentioned last week progressing?" in multi-turn dialogues, the dynamic sensitivity analysis algorithm discovers that although the query weight parameter of a certain attention head currently has a high partial derivative, it has significantly deviated from the initial reference value, and the nonlinear influence factor has greatly reduced its effective sensitivity; at the same time, this parameter has a strong positive correlation with another key projection parameter (interaction weight 0.82), and the optimizer accordingly fine-tunes both synchronously instead of updating them separately, avoiding the destruction of collaborative representation. After iterative verification, the model's accuracy in cross-turn entity referencing is improved without increasing the total number of parameters.

[0076] This embodiment quantifies the importance of parameters by constructing a local sensitivity of component memory performance to model parameters and normalizing it by combining the ratio of the current parameter value to the memory performance. It overcomes the distortion problem of traditional linear sensitivity analysis in non-convex high-dimensional spaces by introducing a nonlinear influence factor based on an exponential decay function to characterize the accelerated decline in memory performance when parameters deviate from the optimal reference value. Furthermore, it models the coupling relationship between parameters using covariance and variance statistics and generates standardized interaction weights to ensure that the optimization process considers the synergistic effect of parameters rather than isolated adjustments. This allows for accurate identification of parameters and their interaction structures that play a key role in memory function, achieving targeted enhancement of memory-related subspaces without increasing model capacity. This significantly improves the stability and generalization ability of large models in tasks such as long-context understanding and knowledge consistency maintenance, and endows the entire memory enhancement process with interpretability, dynamic adaptability, and closed-loop feedback characteristics, avoiding the blindness of empirical parameter tuning and the high computational cost of structural modifications.

[0077] In one embodiment, the dynamic sensitivity factor is calculated using the following formula:

[0078]

[0079] in, This represents the dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration; This represents the memory performance of the i-th component in the t-th iteration, initially... , This represents the initial memory performance of the i-th component; This represents the j-th model parameter value of the i-th component in the t-th iteration; This represents the partial derivative of the component's memory performance with respect to the model parameter values; This represents the relative proportion of model parameters to component memory performance; Indicates the nonlinear influence factor; This represents the reference value of the j-th model parameter of the i-th component; Indicates the attenuation coefficient; This represents the weighted sum of the interaction effects of model parameters; This represents the interaction weight between the j-th and k-th model parameters.

[0080] In this embodiment, the dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration can be a quantitative index calculated in the t-th optimization iteration for the j-th model parameter within the i-th component, comprehensively reflecting its influence on memory performance. It can be used as the basis for adjusting this parameter in the dynamic parameter optimization algorithm, integrating local sensitivity, nonlinear decay, and parameter interaction effects. Specifically, the dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration can be calculated by integrating partial derivatives, relative proportions, nonlinear influence factors, and interactive weighted sums using a formula. The memory performance of the i-th component in the t-th iteration can be a quantitative value of the memory capacity exhibited by the i-th component under the current parameter configuration at the t-th iteration. This value can be used to update the performance benchmark in the sensitivity calculation in real time, supporting dynamic evaluation. For example, the memory performance of the i-th component in the t-th iteration can be recalculated by the monitoring module after each iteration by performing a task based on new parameters and statistically analyzing the memory error rate.

[0081] The initial memory performance of the i-th component can be the baseline value of the component's memory capacity before optimization begins (t=0). It can be used as the initial value for the memory performance in the t-th iteration and participate in the relative proportion calculation. In an exemplary embodiment, the initial memory performance of the i-th component can be obtained from the initial evaluation by the monitoring module in step S100. The j-th model parameter value of the i-th component in the t-th iteration can be the specific value of the j-th learnable parameter in the i-th component at the t-th iteration time. It can be used as the current state input for sensitivity calculation to calculate the degree of deviation and relative proportion.

[0082] The partial derivative of component memory performance with respect to model parameter values ​​can be considered as the first-order local rate of change of component memory performance with respect to specific model parameters. This can be used to characterize the direction and intensity of the direct impact of small parameter variations on memory performance. Furthermore, the partial derivative of component memory performance with respect to model parameter values ​​can be approximated by taking the derivative after fitting the curve to the controlled variables or by using automatic differentiation techniques. The relative ratio of model parameters to component memory performance can be the ratio of the current parameter value to its corresponding component memory performance. This is used to normalize the sensitivity metrics and can eliminate sensitivity bias caused by differences in the absolute magnitude of parameters or performance benchmarks, improving cross-component comparability.

[0083] The reference value of the j-th model parameter of the i-th component can be a baseline configuration value used to measure whether the parameter deviates from the effective memory region. It can also be used as the center point of exponential decay in the nonlinear influence factor to determine the decay start position. In a specific embodiment, the reference value of the j-th model parameter of the i-th component can be one or more of the following, including but not limited to the parameter value at component initialization, the snapshot value when memory performance is optimal on the historical validation set, and the moving average parameter trajectory.

[0084] The decay coefficient can be a hyperparameter controlling the exponential decay rate in nonlinear influence factors. It can be used to adjust the steepness of sensitivity decrease after a parameter deviates from its reference value, balancing exploration and stability. For example, the decay coefficient can include, but is not limited to, a fixed decay coefficient, an adaptive decay coefficient (adjusted during training), and a task-dependent decay coefficient. The weighted sum of the interaction effects of model parameters can be the weighted sum of the interaction weights of the j-th parameter with all other parameters k and their corresponding sensitivity or gradient terms. This can be used to quantify the comprehensive interaction effect of parameter j after considering coupling relationships, correcting isolated sensitivity assessments. The interaction weights between the j-th and k-th model parameters can be the standardized statistical correlation strength between parameters, used to weight the interaction effects. This can be used to reflect the degree of cooperation or competition between parameter pairs (j, k) in the interaction weighted sum. Furthermore, the interaction weights between the j-th and k-th model parameters can be the correlation coefficient obtained by dividing the covariance by their respective standard deviations, or a variant after sparsification.

[0085] Calculating the dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration can be achieved by combining partial derivatives, relative proportions, nonlinear influence factors, and interactive weighted sums according to a formula. Furthermore, calculating the dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration can be achieved through the aforementioned multi-source information fusion mechanism, thereby generating a unified sensitivity index that integrates multi-dimensional information to guide differentiated parameter updates. Obtaining the memory performance of the i-th component in the t-th iteration can be achieved by re-evaluating the component's memory error rate on a verification task through a monitoring module after the t-th parameter update and converting it into a performance score. Furthermore, obtaining the memory performance of the i-th component in the t-th iteration can be achieved through an online evaluation mechanism, thereby providing real-time performance feedback required for dynamic sensitivity calculation. Initializing the component memory performance to the initial memory performance can be achieved by setting the component memory performance to the initial memory performance calculated in step S100 at t=0. In a specific embodiment, initializing the component memory performance to the initial memory performance can be achieved through a pre-evaluation process, thereby ensuring that the sensitivity calculation has an accurate baseline at the optimization starting point.

[0086] Obtaining the j-th model parameter value of the i-th component in the t-th iteration can be achieved by reading the values ​​of the specified component and its parameters from the current model state. For example, obtaining the j-th model parameter value of the i-th component in the t-th iteration can be implemented through a model parameter caching mechanism, thereby providing the current parameter state input to the sensitivity formula. Calculating the partial derivative of the component's memory performance with respect to the model parameter value can be achieved by perturbing the values ​​while keeping other parameters fixed. , observation The changes are then assessed by fitting the local response curve and then differentiating it. Furthermore, the partial derivatives of the component's memory performance with respect to the model parameter values ​​can be calculated analytically using the central difference method or by fitting a surrogate model (such as local linear regression), thereby obtaining the strength of the local causal effect of the parameters on memory performance.

[0087] The relative proportion of model parameters to component memory performance can be calculated by performing a division operation. In an exemplary embodiment, the relative proportion of model parameters to component memory performance can be achieved through dimensionless processing, thereby enabling dimensionless normalization of sensitivity and enhancing cross-parameter comparability. Introducing a nonlinear influence factor to reflect the impact of parameter deviations from reference values ​​can be calculated... This is then incorporated as a multiplicative factor into the sensitivity formula. Furthermore, introducing a nonlinear influence factor to reflect the impact of parameter deviations from the reference value can be achieved through exponential decay function modeling, thereby suppressing parameter sensitivity far from the reference value region and more accurately reflecting its actual contribution. The reference value for the j-th model parameter of the i-th component can be recorded before optimization begins. Alternatively, it can be selected from historical high-performance snapshots. For example, setting the reference value of the j-th model parameter of the i-th component can be achieved through a snapshot storage mechanism, thereby providing an anchor point for nonlinear decay and defining the center of the "effective memory region".

[0088] A decay coefficient is configured to control the nonlinear decay rate. This coefficient can be preset based on task complexity or experience, or dynamically adjusted using an adaptive strategy. In one specific embodiment, the decay coefficient can be configured to control the nonlinear decay rate by fixing... =1.0 is suitable for general scenarios, or The optimization region is gradually tightened with each training epoch, allowing adjustment of the strictness of the nonlinear penalty and balancing robustness and convergence speed. The weighted sum of the interaction effects of model parameters can be calculated through a parameter coupling graph traversal mechanism, explicitly introducing parameter coupling effects into sensitivity evaluation and avoiding isolated optimization. Determining the interaction weights between the j-th and k-th model parameters can be achieved by calculating the covariance and variance based on historical gradients or perturbation response sequences, and then normalizing the correlation coefficient. For example, the interaction weights between the j-th and k-th model parameters can be determined by using the Pearson correlation coefficient as the interaction weights, or by applying a threshold truncation to the correlation coefficient to construct a sparse interaction graph, thereby quantifying the statistical dependence between parameter pairs and supporting the construction of the interaction weighted sum.

[0089] For example, in the context of long-range reasoning tasks in legal documents, the method for increasing the memory capacity of large models in this embodiment could be that, when answering questions such as "According to Article 5 of Chapter 3 and Article 2 of Chapter 7, did the defendant breach the contract?", the system detects the bias parameters of a certain feedforward network layer. The current value has deviated significantly from the initial reference value. Although its partial derivative is high, the nonlinear influence factor reduces its sensitivity to 0.3. Simultaneously, this parameter has a strong negative interaction weight (-0.78) with another attention head weight parameter. After integrating this information, the optimizer applies only a small update to the dynamic sensitivity factor and simultaneously adjusts the associated parameters to maintain synergistic representation. After three iterations, the model's accuracy for joint reasoning across chapter legal clauses improves without increasing the model size.

[0090] This embodiment provides a method to improve the memory capacity of large models. By using the calculation formula of the dynamic sensitivity factor, it integrates the local sensitivity (partial derivative) of component memory performance to individual parameters, the relative proportion of the current state of the parameter to the memory performance, the nonlinear decay effect caused by the parameter deviating from the optimal reference value (modeled by the exponential nonlinear influence factor), and the coupling relationship between parameters (through interactive weighted summation) into a differentiable and optimizable index system. This allows the dynamic sensitivity factor to not only reflect the direct influence of parameters on memory, but also capture the changes in marginal benefits under non-ideal conditions and the synergistic / competitive relationship with other parameters. The parameter adjustment guided by this is no longer globally uniform or experience-driven, but rather a precise intervention for a subset of parameters with high sensitivity, high interaction, and low deviation. This efficiently improves the memory stability of each component without expanding the model structure, achieving fine-grained, dynamic, and closed-loop control of memory capacity. It significantly alleviates problems such as context breakage and knowledge forgetting, enhances the coherence and generalization ability of large models in long sequence tasks, and avoids the computational redundancy and training instability caused by coarse-grained parameter tuning or structural modification in traditional methods.

[0091] In one embodiment, the dynamic parameter optimization and memory capacity iteration improvement algorithm calculates the model parameter adjustment amount based on the dynamic sensitivity factor and component memory performance, and reasonably determines the adjustment range of the model parameters;

[0092] By calculating the unmemorable degree of the current component's memory performance, multiplying the unmemorable degree by the dynamic sensitivity factor, a preliminary adjustment range is obtained as the numerator. The sum of the square of the dynamic sensitivity factor and a very small positive number is introduced into the denominator. At the same time, an adjustment coefficient is introduced to allow control of the adjustment step size according to actual needs.

[0093] The memorability factor can be used to quantify the current memory failure level of a component, and can serve as a driving factor for adjusting the strength of parameters, reflecting the urgency of the component's current memory enhancement. In an exemplary embodiment, the memorability factor can be calculated based on the component's memory performance, typically 1 minus the memory performance or its monotonically increasing form. Furthermore, the memorability factor can include, but is not limited to, one or more of the following: context breakage, knowledge backtracking failure rate, and long-range dependency loss index.

[0094] The model parameter adjustment amount can be the numerical change applied to the model parameters in a single optimization iteration. It can directly determine the direction and magnitude of parameter updates, affecting the improvement effect of component memory performance. In a specific embodiment, the model parameter adjustment amount can be calculated by a dynamic parameter optimization and memory iterative improvement algorithm based on a combination of the numerator, denominator, and adjustment coefficient. The adjustment magnitude can be the absolute value or relative scale of the model parameter adjustment amount, which can be used to control the drasticness of parameter changes and balance optimization speed and stability. The square of the dynamic sensitivity factor can be a non-negative value obtained by multiplying the dynamic sensitivity factor by itself, which can be used to suppress excessive updates of highly sensitive parameters in the denominator and enhance numerical stability. The minimal positive number can be a constant close to zero but greater than zero, which can be used to ensure that the denominator is always positive and guarantee the numerical robustness of the adjustment amount calculation. The numerator can be the dividend part in the parameter adjustment amount calculation formula, which is composed of the non-memoryability and the dynamic sensitivity factor. It can be used to comprehensively reflect the severity of memory defects and parameter influence, driving targeted parameter correction.

[0095] The denominator can be the divisor in the parameter adjustment formula, composed of the square of the dynamic sensitivity factor and a very small positive number. It can be used to apply natural decay to highly sensitive parameters, avoiding parameter oscillations caused by excessive sensitivity. The adjustment coefficient can be a user-configurable hyperparameter, used to scale the overall parameter adjustment amount, allowing flexible control of the optimization step size based on task stage, data complexity, or convergence status. For example, the adjustment coefficient can use a large step size coefficient in the early training phase, a small step size coefficient in the fine-tuning phase, or a dynamic scheduling coefficient. The adjustment step size can be the final update amount applied to the model parameters, determined by both the adjustment coefficient and the basic adjustment magnitude. It can be used to directly influence the convergence behavior and memory performance of the optimization process, improving efficiency.

[0096] The parameter adjustment amount for the model, calculated based on the dynamic sensitivity factor and component memory performance, can be achieved by multiplying the memory deficit by the dynamic sensitivity factor as the numerator, dividing by the denominator consisting of the square of the dynamic sensitivity factor plus a very small positive number, and then multiplying by the adjustment coefficient. Furthermore, this operation can be achieved by using a fixed adjustment coefficient (e.g., 0.1) for stable fine-tuning, or by using an adjustment coefficient that decays with each training round (e.g., cosine annealing) for dynamic convergence control, thereby generating a targeted update amount that matches the degree of memory deficit and parameter sensitivity.

[0097] Determining the appropriate adjustment range for model parameters can be achieved through automatic scaling of the adjustment amount using the numerator-denominator structure. Parameters with high sensitivity but low memory deficit receive smaller updates, while parameters with high deficit and high sensitivity receive moderate enhancements. Furthermore, this operation can be implemented using the aforementioned numerator-denominator construction method, thereby avoiding training instability caused by blindly large updates and achieving refined parameter control.

[0098] The memorability of the current component's memory performance can be calculated by performing a monotonic transformation on the initial memory performance (such as taking the complement or the negative logarithm). Furthermore, this operation can be achieved by setting memorability = 1 - initial memory performance, or setting memorability = -log(initial memory performance + ε), thus transforming the memory performance into an error signal that needs correction, facilitating its participation in optimization calculations.

[0099] Multiplying the memory failure rate by the dynamic sensitivity factor to obtain the initial adjustment range as the numerator can be achieved by performing a scalar multiplication operation, integrating the memory failure intensity and parameter influence. Furthermore, this operation can be implemented through the numerical product of the aforementioned memory failure rate and dynamic sensitivity factor, ensuring that significant updates are triggered only when the component memory is poor and the parameter is sensitive. Introducing the sum of the square of the dynamic sensitivity factor and a very small positive number into the denominator can be achieved by constructing the denominator as (square of the dynamic sensitivity factor + very small positive number). Further, this operation can be implemented by setting the very small positive number to a typical value such as 1e-6, thereby automatically suppressing highly sensitive parameters and preventing excessive update amplitude from causing oscillations. Introducing an adjustment coefficient to control the adjustment step size can be achieved by multiplying the aforementioned calculation result by a configurable scalar coefficient. Further, this operation can be implemented by using a fixed adjustment coefficient (e.g., 0.1) for stable fine-tuning, or by using an adjustment coefficient that decays with training epochs (e.g., cosine annealing) for dynamic convergence control, thereby providing an external intervention interface to adapt to different training stages or task requirements.

[0100] For example, in the scenario of maintaining role consistency in a multi-turn dialogue system, the method to improve the memory capacity of a large model in this embodiment can be as follows: When processing more than ten rounds of dialogue between a user and a virtual character, the monitoring module finds that a certain feedforward network layer frequently forgets the initial character settings (such as occupation and personality) after the 7th round; dynamic sensitivity analysis shows that the bias parameter of this layer is highly sensitive to this type of memory; the algorithm calculates its unmemorability as 0.65 and the dynamic sensitivity factor as 0.8, and substitutes them into the formula to get the numerator = 0.52, the denominator = 0.64 + 1e-6 ≈ 0.64, and the basic adjustment range ≈ 0.8125; if the adjustment coefficient is set to 0.2, the final adjustment step size is 0.1625, and the bias parameter is precisely fine-tuned; subsequent verification shows that the role information retention rate is improved and does not cause a decrease in the performance of other components.

[0101] This embodiment provides a method to improve the memory capacity of large models. Through dynamic parameter optimization and iterative memory capacity enhancement algorithms, the method calculates the adjustment amount of model parameters based on the dynamic sensitivity factor and component memory performance, and rationally determines the adjustment range of model parameters. By calculating the unmemorable degree of the current component memory performance, the unmemorable degree is multiplied by the dynamic sensitivity factor to obtain an initial adjustment range as the numerator. The denominator is the sum of the square of the dynamic sensitivity factor and a very small positive number. Simultaneously, an adjustment coefficient is introduced to control the adjustment step size. By using the unmemorable degree as a quantification driving signal of memory failure, combining it with the dynamic sensitivity factor to construct the numerator to reflect the intensity of correction demand, using the square of the dynamic sensitivity factor and a very small positive number to form the denominator to suppress oscillations of highly sensitive parameters, and flexibly adjusting the final update step size with the adjustment coefficient, this method achieves accurate and robust optimization of key memory-related parameters without changing the model structure. It effectively balances convergence speed and stability, significantly improves the memory coherence and generalization ability of large models in long contexts, multi-turn dialogues, or knowledge-intensive tasks, while maintaining low computational overhead.

[0102] In one embodiment, the formula for calculating the model parameter adjustment amount is as follows:

[0103]

[0104] in, This represents the adjustment amount of the j-th model parameter of the i-th component in the t-th iteration; Indicates the adjustment coefficient; The memorability of the i-th component in the t-th iteration is represented by its unmemorability. This represents the memory performance of the i-th component in the t-th iteration; Indicates the normalized denominator; Represents a very small positive number; This represents the dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration.

[0105] The adjustment amount of the j-th model parameter of the i-th component in the t-th iteration can be the numerical update amount calculated for the j-th specific parameter within the i-th component of the large model in the t-th round of optimization iteration. It can be used as a direct basis for parameter updates, enabling fine-grained and temporal control of memory-related parameters. In an exemplary embodiment, the adjustment amount of the j-th model parameter of the i-th component in the t-th iteration can be one or more of the following, including but not limited to the attention head weight adjustment amount, the feedforward network bias adjustment amount, and the layer normalization scaling factor adjustment amount.

[0106] The unmemorability of the i-th component's memory performance can be a quantitative indicator derived from its current memory performance, representing the degree of memory failure. This indicator can be used to drive parameter adjustment, ensuring a positive correlation between update intensity and the severity of memory problems. Furthermore, the unmemorability of the i-th component's memory performance can be calculated by performing a monotonic transformation (such as taking the complement or negative logarithm) on the memory performance value, thus converting the performance indicator into an error-driven signal to meet the input requirements of the optimization mechanism. The memory performance of the i-th component in the t-th iteration can be the memory capacity assessment value obtained by the i-th component based on statistics from the monitoring module at the t-th iteration time. This value can be used as the basis for calculating the unmemorability, reflecting the component's memory state in the current training / inference phase.

[0107] The normalized denominator can be a term in the parameter adjustment formula, composed of the square of the dynamic sensitivity factor and a minimum positive number. It can be used to adaptively scale the adjustment amount, suppressing excessive updates of highly sensitive parameters and improving numerical stability. In a specific embodiment, the normalized denominator can include, but is not limited to, a high-sensitivity suppression denominator, a low-sensitivity retention denominator, and a numerically stable denominator. Furthermore, to stabilize the parameter adjustment process, a normalized denominator is constructed by using the square of the dynamic sensitivity factor plus a minimum positive number to form the denominator term, automatically applying attenuation to highly sensitive parameters to prevent parameter oscillations or divergence caused by excessive sensitivity. In this embodiment, a minimum positive number is introduced into the normalized denominator to prevent division by zero or numerical instability. A constant close to zero but strictly greater than zero (such as 1e-8) is added to the denominator expression to ensure the mathematical validity of division operations and the robustness of numerical calculations.

[0108] The dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration can be a time-varying sensitivity measure of the influence of the j-th parameter within the i-th component on memory performance in the t-th iteration. It can be used to identify the importance of this parameter to memory function in the current context and determine its weight in the adjustment. Calculating the adjustment amount of the j-th model parameter of the i-th component in the t-th iteration can be done by multiplying the adjustment coefficient by (memorylessness × dynamic sensitivity factor) and then dividing by (dynamic sensitivity factor² + a very small positive number). Furthermore, calculating the adjustment amount of the j-th model parameter of the i-th component in the t-th iteration can be achieved by combining the adjustment coefficient with memorylessness and dynamic sensitivity factor for parameter updates, fusing the three according to the formula structure to form a directional, weighted adjustment amount, thereby generating a parameter update signal with memory orientation, component awareness, and numerical robustness. For example, combining the adjustment coefficient with the memorylessness and dynamic sensitivity factors for parameter updates can be performed independently outside of backpropagation, without relying on the loss gradient, or the adjustment amount can be used as a gradient substitute input to a standard optimizer (such as Adam), thereby decoupling the memory enhancement objective from the task loss optimization and achieving function-oriented parameter evolution.

[0109] For example, in the scenario of understanding cross-chapter citations of legal provisions, the method of improving the memory capacity of a large model in this embodiment can be as follows: When processing questions and answers involving dozens of mutually cited legal provisions, the attention head of a certain Transformer layer is identified by the monitoring module in the 5th iteration as having a 40% error rate in remembering the "priority of higher-level laws" principle; the algorithm calculates its unmemorability as 0.6 and its dynamic sensitivity factor as 0.75 accordingly; substituting into the formula, assuming the adjustment coefficient is 0.15 and the minimum positive number is 1e-8, then the adjustment amount = 0.15 × (0.6 × 0.75) / (0.75² + 1e-8) ≈ 0.12; the system applies this update to the query weight parameters of this attention head; subsequent verification shows that its accuracy in handling questions such as "how this law applies when it conflicts with related laws" is significantly improved, without affecting other semantic understanding capabilities.

[0110] This embodiment provides a method to improve the memory capacity of large models. The method calculates the model parameter adjustment by multiplying the adjustment coefficient by the product of the non-memory factor and the dynamic sensitivity factor, and then dividing by the sum of the square of the dynamic sensitivity factor and a minimum positive number. The non-memory factor reflects the severity of the component's current memory failure, and the dynamic sensitivity factor characterizes the weight of the specific parameter's influence on memory capacity. The product of these two factors forms the basis for the adjustment direction and intensity. The normalized denominator achieves adaptive scaling by adding a minimum positive number to the square of the dynamic sensitivity factor to suppress over-updating of highly sensitive parameters while ensuring that low-sensitivity parameters can still be effectively optimized. The adjustment coefficient provides global step size control to adapt to different training stages or task requirements. This achieves fine-grained, component-aware, and memory-oriented parameter updates, overcoming the limitations of traditional optimizers that rely solely on loss gradients while ignoring memory functionality. Without increasing the number of model parameters or introducing external structures, this method precisely enhances the tuning of key memory parameters, significantly improving the stability and generalization ability of large models in scenarios such as long-context reasoning and knowledge consistency maintenance, while maintaining training efficiency and numerical robustness, achieving an efficient, interpretable, and closed-loop memory enhancement effect.

[0111] In one embodiment, the dynamic parameter optimization and memory iteration improvement algorithm updates the model parameter values ​​based on the current model parameter values ​​and the model parameter adjustment amount. To ensure that the adjusted model parameters are computationally feasible, boundary constraints are introduced. The sum of the current model parameter values ​​and the model parameter adjustment amount is compared with the maximum allowable value of the model parameters, and the smaller value is taken. Then, it is compared with the minimum allowable value of the model parameters, and the larger value is taken as the final updated model parameter value, ensuring that the adjusted parameters do not exceed the limitations of hardware resources or model architecture.

[0112] Here, the current model parameter value can be the actual value of a parameter in the model before the start of this iteration, and can be used as the starting point for parameter updates, participating in the adjustment accumulation operation. The model parameter adjustment amount can be the suggested change amount output by the external optimizer, which is added to the current model parameter value to generate an unconstrained candidate new parameter value. In this embodiment, updating the model parameter value based on the current model parameter value and the model parameter adjustment amount can be achieved by performing an addition operation: the temporary parameter is equal to the sum of the current model parameter value and the model parameter adjustment amount. Furthermore, this operation can be implemented through standard gradient descent, adaptive learning rate methods, or other first-order optimization strategies, thereby generating candidate new parameter values ​​that reflect the optimization direction.

[0113] Boundary constraints can be upper and lower bound mechanisms applied to the range of values ​​of model parameters after update. They can be used to prevent parameters from exceeding the numerical range allowed by hardware representation capabilities or model architecture design due to over-adjustment, ensuring training stability and deployment feasibility. In an exemplary embodiment, boundary constraints can perform a clipping operation after parameter update by pre-setting the maximum and minimum allowable values ​​of model parameters. For example, boundary constraints can include, but are not limited to, one or more of the following: numerical boundaries based on FP16 precision, parameter boundaries based on the input domain of the activation function, and statistical boundaries based on the weight initialization distribution.

[0114] The maximum allowable value of model parameters can be an upper limit determined by hardware precision, activation function saturation region, or training stability requirements. This can serve as an upper bound for boundary constraints, preventing excessively large parameters from causing gradient explosion or numerical overflow. The minimum allowable value of model parameters can be a lower limit determined by hardware precision or model structure requirements. This can serve as a lower bound for boundary constraints, preventing excessively small parameters from causing information loss or underflow. In one specific embodiment, a boundary constraint is introduced by comparing the sum of the current model parameter value and the model parameter adjustment amount with the maximum allowable value of the model parameters, taking the smaller value. This can be achieved by truncating the upper bound of temporary parameters: the temporary parameter equals the smaller of the sum and the maximum allowable value. Furthermore, this operation can be implemented through element-wise comparison and assignment, thereby preventing parameters from exceeding the feasible upper limit due to excessive positive updates.

[0115] The process compares the result from the previous step with the minimum allowable value of the model parameters, taking the larger value as the final updated model parameter value. This can be achieved by truncating the lower bound of the previous result: the updated model parameter value equals the larger of the result and the minimum allowable value. Furthermore, this operation can be implemented through conditional judgment or vectorized comparison, ensuring that the parameters always remain within the legal numerical range. The updated model parameter value can be the final parameter value after boundary constraint processing, which can be used for subsequent forward propagation and performance evaluation. In an exemplary embodiment, the updated model parameter value can be obtained by first calculating the sum of the current value and the adjustment amount, and then comparing and truncating it sequentially with the maximum and minimum allowable values. Hardware resource or model architecture limitations can be external or structural constraints affecting the feasibility of model parameter values, providing a basis for setting boundary constraints and ensuring that the parameter update results can be deployed and executed on the target platform. Ensuring that the adjusted parameters do not exceed the limitations of hardware resources or model architecture can be achieved through the aforementioned two-sided truncation operation to ensure parameter value range compliance, thereby improving the numerical stability and deployment compatibility of the model during training and inference phases.

[0116] The memory performance of the component is further updated based on the updated model parameter values;

[0117] Further updating the component's memory performance based on the updated model parameter values ​​can be achieved by performing forward inference with the new parameters, reassessing the number of memory errors with the assistance of the monitoring module, and calculating the new memory performance. Furthermore, this operation can be implemented by running inference tasks on a validation subset and statistically analyzing the backtracking accuracy of key facts, thereby establishing a parameter-performance mapping feedback to support the next round of optimization.

[0118] The adjusted model parameters directly reflect the changes in component memory performance. The actual change in model parameters in this iteration is calculated by subtracting the model parameter value before adjustment from the updated model parameter value and dividing by the reference value of the model parameter to obtain a dimensionless relative change magnitude, which is then multiplied by the corresponding dynamic sensitivity factor. The non-memory factor is introduced to make the improvement in component memory performance gradually decrease as it approaches the theoretical upper limit.

[0119] The model parameter values ​​before adjustment can be the same as the current model parameter values, referring to the original values ​​before the parameter update operation. These values ​​can be used to calculate the actual change and serve as a benchmark for the relative change magnitude. The reference values ​​for the model parameters can be scale benchmarks used to normalize the parameter changes, typically the absolute value, initial value, or moving average of the parameters. These can be used to eliminate differences in the dimensions of different parameters, making the relative change magnitude comparable across parameters. In a specific embodiment, calculating the actual change in model parameters in this iteration (updated minus pre-adjustment) can be achieved by subtracting the pre-adjustment model parameter values ​​from the updated model parameter values. Furthermore, this operation can be implemented through parameter-by-parameter differencing, thus reflecting the actual effective parameter changes, which may be smaller than the original adjustment amount due to boundary constraints.

[0120] Dividing the actual change in model parameters by the reference value of the model parameters yields the dimensionless relative change magnitude, which can be a standardized update intensity index obtained by dividing the actual change in parameters by the reference value. Furthermore, this operation can be achieved by taking the initial absolute value of the parameters as the reference value or by taking the exponential moving average of the absolute values ​​of the parameters within a sliding window, thus enabling standardization of update intensity across parameter scales. The dimensionless relative change magnitude can measure the relative significance of parameter updates and can be used to couple with dynamic sensitivity factors to evaluate optimization effectiveness. Multiplying the relative change magnitude by the corresponding dynamic sensitivity factor can calculate the coupling index as the relative change magnitude multiplied by the dynamic sensitivity factor. Further, this operation can be achieved by looking up a table or estimating the dynamic sensitivity factor online, thus comprehensively measuring the effectiveness and impact of parameter updates to guide subsequent iteration strategies.

[0121] The theoretical upper limit can be the achievable memory performance limit of a component under a specific task or data distribution. It can serve as a reference point for the non-memory decay mechanism, automatically reducing the optimization intensity as memory performance approaches this value. Introducing non-memory decay causes the improvement in component memory performance to gradually decrease as it approaches the theoretical upper limit. This can be achieved by using non-memory decay as a decay factor in subsequent iterations, applying it to the adjustment amount or coefficient, with its value approaching zero as memory performance nears the theoretical upper limit. Furthermore, this operation can be achieved by equaling non-memory decay to one minus the ratio of the current memory performance to the theoretical upper limit, i.e. Alternatively, it can be implemented as a function whose memorability is equal to a function that decays exponentially in the negative form, thereby forming an adaptive convergence mechanism to avoid ineffective optimization or overfitting in the performance saturation region.

[0122] For example, in the scenario of maintaining fact consistency in a knowledge-intensive question-answering system, the method for improving the memory capacity of a large model in this embodiment can be as follows: When answering questions involving multi-hop reasoning, a certain attention point may cause softmax output saturation due to excessively large parameters, frequently ignoring early facts; the algorithm calculates its adjustment amount as +0.8, but the maximum allowable value of this parameter is 1.5, and the current value is 1.45, so the updated value is truncated to 1.5; then, the verification task is run with the new parameters, and the memory performance increases from 0.62 to 0.68; the relative change is calculated as (1.5-1.45) / 1.0 = 0.05, multiplied by the sensitivity factor of 0.9 to get 0.045; at the same time, since the current memory performance is still far from the theoretical upper limit of 0.85, the unmemorable value is 0.37, allowing for further optimization; if the subsequent performance reaches 0.82, the unmemorable value drops to 0.035, significantly suppressing further adjustments and achieving smooth convergence.

[0123] This embodiment provides a method to improve the memory capacity of large models. It updates model parameter values ​​based on current values ​​and parameter adjustments, introduces boundary constraints to truncate the update results, further updates the memory performance of components based on the updated parameter values, calculates the actual changes in model parameters and normalizes them to dimensionless relative magnitudes, multiplies them by a dynamic sensitivity factor, and introduces non-memoryability as a decay mechanism when memory performance approaches the theoretical upper limit. By embedding boundary constraints determined by hardware resources or model architecture limitations in parameter updates to ensure numerical feasibility, by normalizing parameter changes using reference values ​​and coupling them with the dynamic sensitivity factor to achieve cross-parameter scale optimization effect evaluation, and by automatically decaying the non-memoryability as memory performance approaches the theoretical upper limit to achieve adaptive convergence, this method can efficiently and safely improve the memory performance of large models in key tasks such as long sequence dependencies, contextual coherence, and knowledge backtracking without expanding the model structure. This significantly outperforms traditional empirical parameter tuning or coarse-grained optimization strategies.

[0124] In one embodiment, the dynamic parameter optimization and memory capacity iterative improvement algorithm calculates the memory capacity of the entire large model based on the memory performance of the components after iteration, and evaluates the error between the memory capacity of the entire large model and the target value. If the error is positive, it means that the current memory capacity of the entire large model has not yet reached the target, that is, the effect of the current model parameter adjustment is insufficient, and iterative improvement is needed. If the error is zero or negative, it means that the target has been achieved or exceeded, and the iteration process stops.

[0125] The iterative component memory performance can be the memory performance index obtained after re-evaluation of each component following a round of dynamic parameter optimization. In this embodiment, the iterative component memory performance can be used as input for calculating the current total memory capacity of the system, reflecting the actual impact of this round of parameter adjustments on the component's memory capability.

[0126] The total memory capacity of the entire model can be a quantitative indicator representing the overall memory capability of the current model, derived from the aggregation of component memory performance after iterations. For example, the total memory capacity of the entire model can follow the aggregation rules of the initial memory capacity (such as weighted averaging), but the input is the component performance after the latest round of optimization. Furthermore, the total memory capacity of the entire model can be used to compare with the target value, generating error signals to drive the iterative control logic.

[0127] The target value can be a preset threshold for the total memory capacity of the large model. In an exemplary embodiment, the target value can be used as a convergence criterion for the optimization process, providing a clear direction for memory enhancement. The error can be the difference between the total memory capacity of the large model and the target value (target value minus current memory capacity). In a specific embodiment, the error can be calculated by subtracting values, and the sign determines whether to continue iteration. Furthermore, the error can be used as a basis for determining whether to terminate or continue iteration, achieving closed-loop feedback control.

[0128] Calculating the memory capacity of the entire large model based on the memory performance of the components after iterations can be achieved by applying the same aggregation rule as the initial memory capacity after each round of parameter optimization, merging the updated component memory performance into a global memory capacity. Furthermore, this operation can be implemented using a unified aggregation function, enabling real-time quantification of the current model's memory capacity and providing a benchmark for error calculation. Evaluating the error between the total large model memory capacity and the target value can be done by performing an arithmetic operation of subtracting the current total large model memory capacity from the target value, obtaining a signed error value. Further, this operation can be implemented using a standard numerical comparison module, thereby generating an interpretable optimization deviation signal, clarifying the direction and existence of the gap between the current state and the target.

[0129] Determining whether to continue iteration by checking the sign of the error value: if it's greater than zero, trigger the next round of sensitivity analysis and parameter optimization; otherwise, terminate the process. For example, this judgment logic can be implemented using a conditional branch control structure, thereby establishing an adaptive termination mechanism based on goal achievement and avoiding invalid iterations. Stopping the iteration process when the error is zero or negative can be achieved by interrupting the execution loop of the dynamic parameter optimization and memory-based iterative improvement algorithm when the error is ≤0. Furthermore, this operation can be implemented by the process controller responding to error signals, thereby preventing over-adjustment and wasting computational resources, ensuring that the optimization process converges promptly after the goal is met.

[0130] For example, in scenarios involving cross-paragraph reasoning tasks related to legal provisions, the method for increasing the memory capacity of the large model in this embodiment can be as follows: In question-answering tasks that require comprehensive judgment based on multiple legal provisions, the system presets a target memory capacity of 0.92. After the first round of optimization, the overall memory capacity is 0.85 with a positive error, triggering a second round of sensitivity analysis. This analysis reveals insufficient modeling of the long-term dependence of a certain attention point on the legal provision number. After targeted fine-tuning, the memory capacity increases to 0.93 with an error of -0.01, and the system automatically terminates the iteration. Ultimately, without increasing the number of parameters, the model significantly improves the consistency and accuracy of answers to questions such as "Does this constitute Act Z according to relevant legal provisions and other legal provisions?"

[0131] This embodiment provides a method for improving the memory capacity of a large model. It calculates the memory capacity of the entire large model based on the memory performance of components after iteration and evaluates the error between the total memory capacity and the target value. If the error is positive, iterative improvement continues; if the error is zero or negative, the iteration process stops. By aggregating component-level memory performance into a total memory capacity index and quantifying it against a preset target to generate an error signal, and triggering a new round of parameter optimization or terminating the process based on the error sign, a closed-loop feedback mechanism based on the target value can be introduced. This provides clear convergence criteria for the parameter adjustment process, avoiding over-adjustment or under-adjustment. Without expanding the model structure, it accurately improves the context preservation capability and knowledge backtracking stability in long sequence modeling, thereby significantly enhancing the generalization performance of large models in complex reasoning tasks.

[0132] Furthermore, to achieve the above objectives, the present invention also provides a system for improving the memory capacity of large models, the system comprising:

[0133] The memory monitoring module is used to deploy a monitoring module during model training or inference to record the number of memory errors and the total number of tasks processed for each component of the large model during the task processing phase, and to calculate the initial memory performance of each component based on the number of memory errors and the total number of tasks processed.

[0134] The performance evaluation module is used to calculate the initial memory size of the entire large model based on the initial memory performance of each component, serving as the starting point for dynamic parameter optimization.

[0135] The sensitivity analysis module is used to analyze the relationship between component memory performance and model parameters through dynamic sensitivity analysis algorithms, and to calculate the dynamic sensitivity factor of each component model parameter;

[0136] The parameter optimization module is used to dynamically adjust model parameters based on dynamic sensitivity factors, component memory performance, and model parameters through dynamic parameter optimization and memory capacity iteration improvement algorithms. Combined with sensitivity analysis and iterative verification, it gradually increases the memory capacity of components and the whole machine.

[0137] Other embodiments or specific implementations of the system for improving the memory capacity of large models described in this invention can be found in the above-described method embodiments, and will not be repeated here.

[0138] Furthermore, to achieve the above objectives, the present invention also provides an apparatus for increasing the memory capacity of large models, the apparatus comprising: a memory, a processor, and a program for increasing the memory capacity of large models stored in the memory and executable on the processor, the program for increasing the memory capacity of large models being configured to implement the steps of the method for increasing the memory capacity of large models as described above.

[0139] In addition, to achieve the above objectives, the present invention also provides a medium storing a program for increasing the memory capacity of large models, wherein when the program for increasing the memory capacity of large models is executed by a processor, the program for increasing the memory capacity of large models implements the steps of the method for increasing the memory capacity of large models as described above.

[0140] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for increasing the memory capacity of large models, characterized in that, The method includes: Deploy a monitoring module during model training or inference to record the number of memory errors and the total number of tasks processed for each component of the large model during the task processing phase, and calculate the initial memory performance of each component based on the number of memory errors and the total number of tasks processed. Based on the initial memory performance of each component, the initial memory size of the entire large model is calculated as the starting point for dynamic parameter optimization. The relationship between component memory performance and model parameters is analyzed using a dynamic sensitivity analysis algorithm, and the dynamic sensitivity factor of each component model parameter is calculated. Based on dynamic sensitivity factors, component memory performance, and model parameters, the model parameters are dynamically adjusted through dynamic parameter optimization and memory capacity iteration improvement algorithms. Combined with sensitivity analysis and iterative verification, the memory capacity of components and the whole machine is gradually increased.

2. The method for increasing the memory capacity of large models as described in claim 1, characterized in that, The dynamic sensitivity analysis algorithm evaluates the direct impact of changes in model parameters on the component's memory performance. It obtains the partial derivative of the component's memory performance with respect to the model parameters by taking the derivative of the fitted curve through controlled variable experiments. The partial derivative is then multiplied by the ratio of the component's model parameter value to its memory performance. Introducing a nonlinear influence factor, we describe the impact of model parameters deviating from the reference value using an exponential decay function, simulating the nonlinear effect on component memory performance when model parameters deviate from the optimal state during actual training. Considering the interaction between model parameters, the correlation between each model parameter and other model parameters is calculated using statistical methods. The covariance and variance between model parameters are calculated to obtain a standardized interaction weight.

3. The method for increasing the memory capacity of large models as described in claim 2, characterized in that, The formula for calculating the dynamic sensitivity factor is as follows: in, This represents the dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration; This represents the memory performance of the i-th component in the t-th iteration, initially... , This represents the initial memory performance of the i-th component; This represents the j-th model parameter value of the i-th component in the t-th iteration; This represents the partial derivative of the component's memory performance with respect to the model parameter values; This represents the relative proportion of model parameters to component memory performance; Indicates the nonlinear influence factor; This represents the reference value of the j-th model parameter of the i-th component; Indicates the attenuation coefficient; This represents the weighted sum of the interaction effects of model parameters; This represents the interaction weight between the j-th and k-th model parameters.

4. The method for increasing the memory capacity of large models as described in claim 1, characterized in that, The dynamic parameter optimization and memory capacity iteration improvement algorithm calculates the model parameter adjustment amount based on the dynamic sensitivity factor and component memory performance, and reasonably determines the adjustment range of the model parameters. By calculating the unmemorable degree of the current component's memory performance, multiplying the unmemorable degree by the dynamic sensitivity factor, a preliminary adjustment range is obtained as the numerator. The sum of the square of the dynamic sensitivity factor and a very small positive number is introduced into the denominator. At the same time, an adjustment coefficient is introduced to allow control of the adjustment step size according to actual needs.

5. The method for increasing the memory capacity of large models as described in claim 4, characterized in that, The formula for calculating the model parameter adjustment amount is as follows: in, This represents the adjustment amount of the j-th model parameter of the i-th component in the t-th iteration; Indicates the adjustment coefficient; The memorability of the i-th component in the t-th iteration is represented by its unmemorability. This represents the memory performance of the i-th component in the t-th iteration; Indicates the normalized denominator; Represents a very small positive number; This represents the dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration.

6. The method for increasing the memory capacity of large models as described in claim 4, characterized in that, The dynamic parameter optimization and memory iteration improvement algorithm updates the model parameter values ​​based on the current model parameter values ​​and the model parameter adjustment amount. To ensure that the adjusted model parameters are computationally feasible, boundary constraints are introduced. The sum of the current model parameter value and the model parameter adjustment amount is compared with the maximum allowable value of the model parameter, and the smaller value is taken. Then, it is compared with the minimum allowable value of the model parameter, and the larger value is taken as the final updated model parameter value. This ensures that the adjusted parameters do not exceed the limitations of hardware resources or model architecture. The memory performance of the component is further updated based on the updated model parameter values; The adjusted model parameters directly reflect the changes in component memory performance. The actual change in model parameters in this iteration is calculated by subtracting the model parameter value before adjustment from the updated model parameter value and dividing by the reference value of the model parameter to obtain a dimensionless relative change magnitude, which is then multiplied by the corresponding dynamic sensitivity factor. By introducing non-memory properties, the improvement in component memory performance gradually decreases as it approaches the theoretical upper limit.

7. The method for increasing the memory capacity of large models as described in claim 1, characterized in that, The dynamic parameter optimization and memory capacity iterative improvement algorithm calculates the memory capacity of the entire large model based on the memory performance of the components after iteration, and evaluates the error between the memory capacity of the entire large model and the target value. If the error is positive, it means that the current memory capacity of the entire large model has not yet reached the target, that is, the effect of the current model parameter adjustment is insufficient, and iterative improvement is needed. If the error is zero or negative, it means that the target has been achieved or exceeded, and the iteration process stops.

8. A system for improving the memory capacity of large models, characterized in that, The system includes: The memory monitoring module is used to deploy a monitoring module during model training or inference to record the number of memory errors and the total number of tasks processed for each component of the large model during the task processing phase, and to calculate the initial memory performance of each component based on the number of memory errors and the total number of tasks processed. The performance evaluation module is used to calculate the initial memory size of the entire large model based on the initial memory performance of each component, serving as the starting point for dynamic parameter optimization. The sensitivity analysis module is used to analyze the relationship between component memory performance and model parameters through dynamic sensitivity analysis algorithms, and to calculate the dynamic sensitivity factor of each component model parameter; The parameter optimization module is used to dynamically adjust model parameters based on dynamic sensitivity factors, component memory performance, and model parameters through dynamic parameter optimization and memory capacity iteration improvement algorithms. Combined with sensitivity analysis and iterative verification, it gradually increases the memory capacity of components and the whole machine.

9. A device for increasing the memory capacity of large models, characterized in that, The device includes: a memory, a processor, and a program for increasing the memory capacity of large models stored in the memory and executable on the processor, the program for increasing the memory capacity of large models being configured to implement the steps of the method for increasing the memory capacity of large models as claimed in any one of claims 1 to 7.

10. A medium, characterized in that, The medium stores a program for increasing the memory capacity of large models, which, when executed by a processor, implements the steps of the method for increasing the memory capacity of large models as described in any one of claims 1 to 7.

Images