Method and system for improving memory capacity of large model

Abstract

The application relates to the technical field of large models, in particular to a method and system for improving the memory capacity of a large model, which deploys a monitoring module in the training or reasoning process, counts the memory error number and total task quantity of each component, and calculates the initial memory performance of the component; based on the performance evaluation result of each component, the initial memory capacity of the whole model is quantified as the optimization starting point; the influence degree of each component parameter on the memory performance is mined by using a dynamic sensitivity analysis algorithm, and a dynamic sensitivity factor is calculated; the model parameters are adaptively adjusted by combining the sensitivity factor, the current parameter state and the memory performance through a dynamic parameter optimization and iteration improvement algorithm, and an iteration verification mechanism is introduced to optimize the memory capacity of each component and the whole machine in a closed loop. The method realizes fine quantification and parameter-level dynamic regulation of the memory performance of the model, effectively improves the memory stability and long-term dependence processing capacity of the large model in knowledge retention, context understanding and other tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large models, in particular to a method and system for improving the memory capacity of a large model. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, large models are increasingly widely used in natural language processing, computer vision, speech recognition and other fields. The continuous increase in model size brings stronger expression and reasoning capabilities, but at the same time puts higher requirements on the memory capacity of the model. The memory capacity of a large model refers to the ability of the model to effectively store and accurately call historical information, knowledge parameters and context associations during training and reasoning. However, in actual applications, due to problems such as parameter redundancy, gradient sparsity, attention mechanism bias and training data noise, large models often exhibit forgetting or memory errors, such as context understanding bias, inaccurate knowledge recall, and long sequence dependency breakage, which seriously affect the stability and generalization performance of the model.

[0003] Existing methods for improving the memory capacity of a model mostly focus on structural adjustment (such as introducing external memory units), optimizing training strategies (such as continuous learning, replay mechanism) or increasing model capacity, but these methods often bring problems such as dramatic increase in computational overhead, unstable training or diminishing marginal benefits. At the same time, existing parameter tuning processes mostly rely on empirical tuning or global optimization algorithms, lack fine-grained modeling of the memory performance differences of different components (such as attention heads, feedforward networks, layer normalization modules, etc.), and do not fully consider the dynamic impact of each model parameter on memory function and their mutual coupling relationship.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a method and system for improving the memory capacity of a large model, aiming to solve the technical problems that existing large models lack quantitative analysis of the memory performance of each component and parameter-level dynamic optimization mechanism, resulting in memory capacity improvement being limited by empirical tuning and overall structural modification, making it difficult to achieve efficient and accurate memory capacity enhancement.

[0006] To achieve the above purpose, the present application provides a method for improving the memory capacity of a large model, comprising:

[0007] Deploying a monitoring module during model training or reasoning, recording the number of memory errors and the total number of task processing of each component of the large model during the task processing stage, calculating the initial memory performance of each component according to the number of memory errors and the total number of task processing;

[0008] Calculate the initial memory amount of the whole large model based on the initial memory performance of each component as the starting point of dynamic parameter optimization;

[0009] Analyze the relationship between the component memory performance and the model parameters through a dynamic sensitivity analysis algorithm, and calculate the dynamic sensitivity factor of each component model parameter;

[0010] Based on the dynamic sensitivity factor, component memory performance, and model parameters, dynamically adjust the model parameters through a dynamic parameter optimization and memory amount iterative improvement algorithm, and gradually improve the memory amount of the components and the whole machine by combining sensitivity analysis and iterative verification.

[0011] Optionally, the dynamic sensitivity analysis algorithm evaluates the direct impact of model parameter changes on component memory performance, fits a curve by controlling variable experiments, and obtains the partial derivative of component memory performance with respect to model parameters by taking the derivative; the partial derivative is multiplied by the ratio of component model parameter value to component memory performance;

[0012] A nonlinear influence factor is introduced again to describe the influence when the model parameters deviate from the reference value through an exponential decay function, simulating the nonlinear effect of model parameters deviating from the optimal state on component memory performance in actual training;

[0013] Consider the interaction between model parameters, calculate the correlation between each model parameter and other model parameters through statistical methods, calculate the covariance and variance between model parameters, and obtain a standardized interaction weight.

[0014] Optionally, the calculation formula of the dynamic sensitivity factor is:

[0015]

[0016] wherein, represents the dynamic sensitivity factor of the jth model parameter of the ith component in the tth iteration; represents the memory performance of the ith component in the tth iteration, initially , represents the initial memory performance of the ith component; represents the jth model parameter value of the ith component in the tth iteration; represents the partial derivative of component memory performance with respect to model parameter value; represents the relative proportion of model parameters to component memory performance; represents the nonlinear influence factor; represents the reference value of the jth model parameter of the ith component; represents the decay coefficient; represents the weighted sum of the interaction between model parameters; represents the interaction weight between the jth and kth model parameters.

[0017] Optionally, the dynamic parameter optimization and memory amount iterative promotion algorithm calculates the model parameter adjustment amount according to the dynamic sensitivity factor and the component memory performance calculation model parameter, and reasonably determines the adjustment range of the model parameter;

[0018] By calculating the non-memorability of the current component memory performance, the non-memorability is multiplied by the dynamic sensitivity factor to obtain a preliminary adjustment range as the numerator, and the square of the dynamic sensitivity factor and a small positive number are introduced in the denominator; At the same time, a regulation coefficient is introduced to allow the step of adjustment to be controlled according to actual needs.

[0019] Optionally, the calculation formula of the model parameter adjustment amount is as follows:

[0020]

[0021] Among them, represents the jth model parameter adjustment amount of the ith component in the tth iteration; represents the regulation coefficient; represents the non-memorability of the memory performance of the ith component in the tth iteration; represents the memory performance of the ith component in the tth iteration; represents the normalized denominator; represents a small positive number; represents the dynamic sensitivity factor of the jth model parameter of the ith component in the tth iteration.

[0022] Optionally, the dynamic parameter optimization and memory amount iterative promotion algorithm updates the model parameter value based on the current model parameter value and the model parameter adjustment amount. In order to ensure that the adjusted model parameter is feasible in calculation, a boundary constraint is introduced. The sum of the current model parameter value and the model parameter adjustment amount is compared with the maximum allowed value of the model parameter, and the smaller value of the two is taken. Then, it is compared with the minimum allowed value of the model parameter, and the larger value of the two is taken as the final updated model parameter value, so as to ensure that the adjusted parameter will not exceed the limit of hardware resources or model architecture;

[0023] Further update the memory performance of the component based on the updated model parameter value;

[0024] The actual model parameter change amount in this iteration is calculated by directly reflecting the change of the memory performance of the component through the adjusted model parameter, that is, the updated model parameter value minus the adjusted model parameter value, divided by the reference value of the model parameter, to obtain a dimensionless relative change range, which is multiplied by the corresponding dynamic sensitivity factor. The non-memorability is introduced to gradually reduce the component memory performance promotion range when it approaches the theoretical upper limit.

[0025] Optionally, the dynamic parameter optimization and memory quantity iterative improvement algorithm calculates the memory quantity of the entire large model based on the memory performance of the components after iteration, and evaluates the error between the memory quantity of the entire large model and the target value. If the error is positive, it means that the current memory quantity of the entire large model has not reached the target, that is, the effect of the current model parameter adjustment is insufficient, and iteration needs to be continued to improve. If the error is zero or negative, it means that the target has been reached or exceeded, and the iteration process is stopped.

[0026] In addition, to achieve the above-mentioned purpose, the application also provides a system for improving the memory quantity of a large model, the system comprising:

[0027] a memory monitoring module, configured to deploy a monitoring module in the model training or inference process, record the number of memory errors and the total number of task processing of each component of the large model in the task processing stage, and calculate the initial memory performance of each component according to the number of memory errors and the total number of task processing;

[0028] a performance evaluation module, configured to calculate the initial memory quantity of the entire large model based on the initial memory performance of each component, as the starting point of dynamic parameter optimization;

[0029] a sensitivity analysis module, configured to analyze the relationship between the component memory performance and the model parameters by a dynamic sensitivity analysis algorithm, and calculate the dynamic sensitivity factor of each component model parameter;

[0030] a parameter optimization module, configured to dynamically adjust the model parameters based on the dynamic sensitivity factor, the component memory performance and the model parameters by a dynamic parameter optimization and memory quantity iterative improvement algorithm, and gradually improve the memory quantity of the components and the entire machine by combining sensitivity analysis and iterative verification.

[0031] In addition, to achieve the above-mentioned purpose, the application also provides a device for improving the memory quantity of a large model, the device comprising a memory, a processor, and a program for improving the memory quantity of a large model stored on the memory and executable on the processor, the program for improving the memory quantity of a large model being configured to implement the steps of the method for improving the memory quantity of a large model according to any one of the above.

[0032] In addition, to achieve the above-mentioned purpose, the application also provides a medium having a program for improving the memory quantity of a large model stored thereon, the program for improving the memory quantity of a large model being executable by a processor to implement the steps of the method for improving the memory quantity of a large model according to any one of the above.

[0033] The application provides a method for improving the memory capacity of a large model, which realizes fine quantification and evaluation of the memory performance of each component of the large model by deploying a monitoring module, accurately identifies the influence degree of key parameters on the memory capacity by combining a dynamic sensitivity analysis algorithm, and then adaptively adjusts the model parameters by using a dynamic parameter optimization and iteration improvement mechanism, thereby effectively improving the memory capacity of the components and the overall model without increasing the model size. The method realizes the explainability, dynamics and closed-loop optimization of memory enhancement, significantly improves the stability and generalization capability of the large model in long sequence modeling, knowledge retention and context understanding, and at the same time avoids the high cost and low efficiency problems caused by the dependence of traditional methods on experience parameter adjustment or structure modification. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a flowchart of an embodiment of the method for improving the memory capacity of a large model.

[0035] The implementation of the object, functional characteristics and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0036] It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0037] Reference Figure 1 , Figure 1 is a flowchart of an embodiment of the method for improving the memory capacity of a large model, which proposes an embodiment of the method for improving the memory capacity of a large model.

[0038] In an embodiment, the method for improving the memory capacity of a large model comprises:

[0039] Step S100, deploying a monitoring module in the model training or inference process, recording the number of memory errors and the total number of task processing of each component of the large model in the task processing stage, and calculating the initial memory performance of each component according to the number of memory errors and the total number of task processing.

[0040] The monitoring module can be a software or logic unit embedded in the model training or inference process for collecting and recording component-level memory behavior data, which can be used to realize fine-grained statistics of the number of memory errors and the total number of task processing of each component of the large model in the real task flow, and provide a data basis for initial memory performance evaluation. In this embodiment, the monitoring module can insert a hook or a callback function in the forward propagation or backward propagation path of the model to capture the input, output and task result of each component in real time, and compare the expected and actual responses to identify memory errors. Exemplarily, the monitoring module can include but is not limited to one or more of an attention head-based monitor, a feedforward network layer-based monitor, a layer normalization module-based monitor, etc.

[0041] It should be understood that the present memory system is divided into two processing modes, one is short-term memory, and the other is long-term memory. For short-term memory, the text model is used to extract entities, intents, and constraint conditions in the dialogue, and the memory volume is compressed. For long-term memory, the recent information is returned first during retrieval, and the related entities are associated through the knowledge graph to avoid confusion with the same name. The entity recognition model for processing short-term memory needs to extract entities, intents, constraint conditions, etc. The bert-base / chinese basic model is used for transfer training. After sorting the training data of Chinese entities, intents, and constraint conditions, transfer training is performed, and the final model has low memory occupation and far exceeds the speed of large models. The extracted data is sent to the large model together with the dialogue history and new questions. The data volume is much smaller than simply splicing the dialogue history data. When processing long-term memory, the time series database Prometheus is used to save the dialogue history data. By taking advantage of the characteristics of the time series database, the weight of recent information is increased, and recent information can be returned first. Based on the BERT-Base / Large basic text model, the Chinese entity association training set is sorted, and an entity association model that can extract entity associations is trained. By extracting the association relationship, the data volume in the dialogue memory is greatly reduced.

[0042] Each component of the large model can be a basic functional unit that constitutes the large model, such as an attention head, a feedforward network, a layer normalization module, etc., which can be used to undertake specific computing tasks and participate in information storage and retrieval in context processing, and its performance directly affects the overall memory capability. Further, each component of the large model can include, but is not limited to, a single attention head in a multi-head attention mechanism, a feedforward neural network in a Transformer block, a layer normalization unit in a residual connection, etc.

[0043] The number of memory errors can be the number of failures of the component to correctly call or maintain historical information, knowledge parameters, or context associations when processing tasks, which can be used as a negative indicator to measure the reliability of the component memory and to calculate the initial memory performance. In an exemplary embodiment, the number of memory errors can be determined by the monitoring module by comparing the model output with the standard answer or consistency constraint condition. The total number of task processing can be the total number of task instances handled by the component within a certain time period or data batch, which can be used as a normalization benchmark for the number of memory errors to ensure the fairness and comparability of memory performance evaluation. The initial memory performance can be a quantitative indicator of the component-level memory capability calculated based on the number of memory errors and the total number of task processing, which can be used to reflect the memory stability of the component under the current parameter configuration as the starting point for subsequent sensitivity analysis and parameter optimization. In a specific embodiment, the initial memory performance can be calculated in the form of accuracy, error rate inverse, or weighted success rate, etc.

[0044] The monitoring module can be deployed in the model training or inference process, and the monitoring logic can be embedded in the model execution process to synchronously collect the behavior data of the data stream through each component. Further, the monitoring module can be deployed in the model training or inference process by integrating the monitoring logic into the model calculation graph, so that the observability of the memory behavior of each component can be established, and the original data support for subsequent quantitative evaluation can be provided.

[0045] The number of memory errors and the total number of task processing of each component of the large model in the task processing stage can be recorded, that is, for each task instance, the monitoring module compares the component output with the expected memory behavior, and accumulates the error and the total processing times. Further, the number of memory errors and the total number of task processing of each component of the large model in the task processing stage can be recorded by synchronously counting in the verification task stream, so that standardized statistical data that can be used for performance evaluation can be generated, and fine-grained memory capability diagnosis can be supported.

[0046] The initial memory performance of each component can be calculated according to the number of memory errors and the total number of task processing, that is, the original statistical data can be converted into performance scores by using a ratio, success rate or weighted index formula. Further, the initial memory performance of each component can be calculated according to the number of memory errors and the total number of task processing by using an online sliding window statistics or batch aggregation method, so that discrete error events can be converted into continuous and comparable memory performance metrics.

[0047] Step S200, based on the initial memory performance of each component, the initial memory amount of the entire large model is calculated as the starting point of dynamic parameter optimization.

[0048] The initial memory amount of the entire large model can be a global memory capability metric aggregated from the initial memory performance of all components, which can be used to represent the overall memory level of the model and provide an initial reference value for dynamic parameter optimization. In an exemplary embodiment, the initial memory amount of the entire large model can be integrated by weighted average, maximum pooling or component importance-based fusion strategy.

[0049] The initial memory performance of all components can be integrated by a preset aggregation rule, that is, the initial memory performance of each component can be fused by weighted average and the like. Further, the initial memory performance of all components can be integrated by giving higher weight to frequently used components, so that a global memory capability baseline can be formed to measure the subsequent optimization progress.

[0050] Step S300, the relationship between the component memory performance and the model parameters is analyzed by a dynamic sensitivity analysis algorithm, and the dynamic sensitivity factor of each component model parameter is calculated.

[0051] The dynamic sensitivity analysis algorithm can be an adaptive analysis method for quantifying the influence of model parameter variation on the memory performance of the component, and can be used to accurately identify parameters that play a key role in memory capability and their dynamic influence strength, breaking through the limitations of static or global sensitivity analysis. In one specific embodiment, the dynamic sensitivity analysis algorithm can introduce local parameter perturbation during training or inference, observe the change in memory performance, and combine gradient information or proxy model fitting parameter-performance mapping relationship. For example, the dynamic sensitivity analysis algorithm can include but is not limited to local sensitivity analysis based on gradient, black-box sensitivity estimation based on proxy model, perturbation response analysis based on task playback, etc.

[0052] The model parameters can be learnable numerical variables in the large model, including weight matrix, bias vector, normalization parameter, etc., which can be used to determine the mapping capability and internal representation structure of the model, and the configuration directly affects the storage and retrieval efficiency of the memory information. The dynamic sensitivity factor can be a time-varying numerical index representing the influence of a specific model parameter on the memory performance of the component, which can be used to identify which parameters have a high influence on memory function and guide the priority and direction of subsequent parameter adjustment. In one exemplary embodiment, the dynamic sensitivity factor can be calculated by the response relationship between parameter perturbation and memory performance change. For example, the dynamic sensitivity factor can include but is not limited to attention weight sensitivity factor, feedforward network bias sensitivity factor, layer normalization scaling factor sensitivity factor, etc.

[0053] By analyzing the relationship between the component memory performance and the model parameters through the dynamic sensitivity analysis algorithm, a small perturbation can be applied in the parameter space, and the corresponding component memory performance change can be observed to establish a mapping model. Further, the relationship between the component memory performance and the model parameters analyzed by the dynamic sensitivity analysis algorithm can be achieved by local linear approximation or non-parametric regression, thereby revealing the causal effect of the parameters on the memory function and providing a basis for directed optimization.

[0054] Calculating the dynamic sensitivity factor of each component model parameter can be based on the ratio of parameter perturbation amplitude to memory performance change or fitting coefficient to determine the sensitivity value. Further, calculating the dynamic sensitivity factor of each component model parameter can be achieved by sliding window historical performance difference or proxy model gradient estimation, thereby quantifying the contribution of each parameter to the memory capability and distinguishing between key and redundant parameters.

[0055] Step S400, based on the dynamic sensitivity factor, the component memory performance and the model parameters, the model parameters are dynamically adjusted through the dynamic parameter optimization and memory capacity iterative improvement algorithm, and the sensitivity analysis and iterative verification are combined to gradually improve the memory capacity of the component and the whole machine.

[0056] The dynamic parameter optimization and memory capacity iterative improvement algorithm can be a closed-loop optimization mechanism that combines sensitivity factors and memory performance feedback to adaptively adjust model parameters to improve memory capacity. It can be used to fine-tune and adaptively adjust key memory-related parameters without increasing the model size, continuously improving the memory capacity of components and the entire machine. In a specific embodiment, the dynamic parameter optimization and memory capacity iterative improvement algorithm can apply targeted updates to high-impact parameters based on dynamic sensitivity factors in each iteration, and evaluate the memory capacity improvement effect through validation tasks to form a feedback loop. For example, the dynamic parameter optimization and memory capacity iterative improvement algorithm can include but is not limited to sensitivity-weighted gradient descent optimization, memory performance-driven parameter reinitialization strategy, and memory-oriented fine-tuning algorithm for multi-component collaboration.

[0057] Applying a larger update step or a specific optimization strategy to high-impact parameters based on sensitivity factors while preserving low-sensitivity parameters can be updating parameters after weighting gradients by sensitivity factors in backpropagation. Further, applying a larger update step or a specific optimization strategy to high-impact parameters based on sensitivity factors while preserving low-sensitivity parameters can be achieved by training only the high-sensitivity parameter subset during the fine-tuning phase, thereby enabling differential control of parameter updates and improving the optimization efficiency of memory-related parameters.

[0058] Re-running the monitoring module to evaluate memory performance after each round of parameter adjustment forms an analysis-optimization-validation closed loop, which can be running long sequence tasks on the validation set to test the context retention ability. Further, re-running the monitoring module to evaluate memory performance after each round of parameter adjustment forms an analysis-optimization-validation closed loop, which can be achieved by checking the accuracy of historical information retrieval through knowledge backtracking tasks, thereby ensuring the stability and generalization of memory capacity improvement and avoiding overfitting or performance shocks.

[0059] Taking a long document question answering system as an example, the method for improving the memory capacity of a large model in this embodiment can record the number of errors of each attention head of the Transformer when answering cross-chapter questions when processing a question answering task involving a multi-chapter technical manual; dynamic sensitivity analysis finds that the bias parameters of certain feedforward network layers are highly sensitive to inter-chapter reference resolution; the dynamic parameter optimization algorithm accordingly fine-tunes these parameters, and in subsequent iterations, verifies the improvement in the accuracy of the model's answers to questions such as "Is the method mentioned in Chapter 3 applicable to the scenario in Chapter 5?", thereby enhancing long-distance dependency modeling capability without increasing the number of model layers or the total number of parameters.

[0060] The embodiment realizes quantitative evaluation of the component-level memory performance by deploying a monitoring module to perform fine-grained statistics on memory errors of each component of the large model in a real task flow, breaking through the limitations of traditional methods that only rely on overall indicators or structural modification; on this basis, a dynamic sensitivity analysis algorithm is introduced to accurately depict the dynamic influence degree of different model parameters on the component memory capacity, solving the problem that existing parameter tuning strategies ignore the parameter-memory coupling relationship; further, the dynamic parameter optimization and memory quantity iterative improvement algorithm is used to take the sensitivity factor as the optimization guide, realizing adaptive and closed-loop adjustment of key memory-related parameters without expanding the model size, which can not only improve the memory stability of a single component, but also enhance the generalization ability of the whole machine in long sequence dependence modeling, context consistency maintenance and knowledge backtracking through collaborative optimization between components, while avoiding the blindness of experience tuning and the computational overhead caused by external memory structure, thereby achieving the technical effects of efficient, interpretable and dynamic evolution of memory enhancement.

[0061] In one of the embodiments, the dynamic sensitivity analysis algorithm obtains the partial derivative of the component memory performance with respect to the model parameter by evaluating the direct influence of the model parameter change on the component memory performance, fitting a curve by controlling variable experiments, and taking the derivative; the partial derivative is multiplied by the ratio of the component model parameter value to the component memory performance;

[0062] The evaluation of the direct influence of the model parameter change on the component memory performance can be to disturb only the target parameter while keeping other parameters unchanged, and to observe the change in the component memory performance. Further, this operation can be realized by sampling multiple points in the parameter neighborhood, fitting the local response curve of the memory performance with respect to the parameter, and then taking the analytical or numerical derivative, so that the local sensitivity of the parameter to the memory performance can be obtained as the influence strength index under linear approximation. Illustratively, the fitting of the local response curve can use polynomial or Gaussian process regression to fit the local response surface and then take the derivative, or use finite difference method to directly calculate the performance difference quotient under forward and backward disturbance.

[0063] Multiplying the partial derivative by the ratio of the component model parameter value to the component memory performance can be to calculate the product of the partial derivative and the ratio of the current value of the parameter to the current memory performance for each parameter. In a specific embodiment, this operation is realized by calculating (∂M / ∂θ) × (θ / M), where M is the component memory performance and θ is the parameter value, thereby realizing relative normalization of the sensitivity and eliminating evaluation bias caused by parameter magnitude or performance absolute value difference.

[0064] A nonlinear influence factor is further introduced to describe the influence of the model parameter deviating from the reference value through an exponential decay function, simulating the nonlinear effect of the model parameter deviating from the optimal state on the component memory performance in actual training;

[0065] The nonlinear influence factor can be an adjustment coefficient for describing the nonlinear attenuation effect of the model parameter deviating from the reference value on the memory performance of the component, and can be used to introduce a penalty mechanism for the degree of parameter deviation in the sensitivity calculation, so that the parameters far from the optimal region obtain a lower effective sensitivity, and the actual influence on the memory performance is more truly reflected. In the embodiment, the nonlinear influence factor can be constructed by an exponential decay function exp(-a |0-0|), where 0 is the current parameter value, 0 is the reference value, and a is the decay rate control parameter. Further, the nonlinear influence factor can include but is not limited to one or more of an L1 distance-based decay factor, an L2 distance-based decay factor, an adaptive decay factor based on the curvature of the task loss, and the like.

[0066] The exponential decay function can be a form of a monotonically decreasing nonlinear function with the independent variable increasing, which is commonly used to model the influence relationship that rapidly decays with the increase of the deviation degree, and can be used to quantify the inhibitory effect of the parameter deviating from the reference value on the memory performance, and reflect the nonlinear degradation of the memory ability caused by the parameter misalignment in the training process. Exemplarily, the exponential decay function can adopt a form of one-sided exponential decay, bilateral symmetric exponential decay, or piecewise exponential decay, etc.

[0067] The reference value can be a benchmark point for measuring whether the current state of the model parameter is in the effective memory region, and usually corresponds to the historical optimal or initialized configuration, which can be used as an anchor point for the calculation of the nonlinear influence factor to determine the starting position of the parameter deviation. In an exemplary embodiment, the reference value can include a parameter snapshot at the beginning of training, a parameter value corresponding to the peak memory performance on the validation set, or a parameter mean value in a sliding window, etc.

[0068] The model parameter deviating from the optimal state can be the difference between the current model parameter and the ideal parameter configuration that can maximize the memory performance of the component, which can be used to trigger the condition of the nonlinear effect and identify the parameter configuration region that can cause a significant decrease in memory ability. The nonlinear effect can be a phenomenon of the model parameter change having a non-proportional and non-constant rate of influence on the memory performance of the component, which can be used to explain why small parameter perturbations have a dramatic effect in some regions and a weak effect in other regions, and improve the authenticity of the sensitivity analysis. By introducing a nonlinear influence factor and describing the influence of the model parameter deviating from the reference value through an exponential decay function, the sensitivity can be automatically attenuated with the parameter deviating from the optimal region by using the L1 norm to measure the degree of deviation to enhance sparsity, or using the L2 norm to smooth the decay curve, so as to be more consistent with the non-convex performance surface characteristics in the actual training. The simulation of the nonlinear effect of the model parameter deviating from the optimal state on the memory performance of the component in the actual training can be to embed a nonlinear attenuation term in the sensitivity calculation, so that the parameters far from the reference value obtain a lower effective sensitivity even if the partial derivative is large, so as to avoid excessive reliance on the local linear approximation in the suboptimal region, and improve the optimization stability.

[0069] The interaction between the model parameters is considered, the correlation between each model parameter and other model parameters is calculated by a statistical method, the covariance and variance between the model parameters are calculated, and a normalized interaction weight is obtained.

[0070] The interaction between the model parameters can be a coupling or a dependent relationship between different model parameters when jointly affecting the performance of component memory, and can be used to reveal the synergistic or counteracting effects that isolated parameter adjustment can ignore, thereby providing a basis for joint optimization. The statistical method can be a set of mathematical tools for inferring the relationship between parameters from observation data, and can be used to support the calculation of correlation, covariance and other indicators, and provide a data basis for the construction of interaction weight. In a specific embodiment, the statistical method can include Pearson correlation coefficient estimation, Spearman rank correlation analysis, local covariance estimation based on sliding window, etc. The correlation can be the statistical association degree between two model parameters in the process of affecting the performance of component memory, and can be used to preliminarily identify the parameter pairs that may have an interaction.

[0071] The covariance can be a second-order statistical quantity that measures the joint variation trend of two model parameters in the gradient direction of memory performance, and can be used to quantify the joint influence strength of the synergistic variation between parameters on memory performance. The variance can be a measure of the degree of fluctuation of a single model parameter in the response of memory performance, and can be used to normalize the covariance and eliminate the influence of dimension. The normalized interaction weight can be a dimensionless parameter interaction strength indicator obtained by dividing the covariance by the respective standard deviation (i.e., the correlation coefficient) or other normalization methods, and can be used to adjust the amplitude of multi-parameter joint update in dynamic parameter optimization, avoiding optimization shock or convergence failure caused by ignoring coupling relationship. In this embodiment, the normalized interaction weight can be obtained by calculating the covariance matrix of the parameter pairs and normalizing it according to the variance to form a symmetric interaction weight matrix. For example, the normalized interaction weight can include Pearson correlation weight, partial correlation interaction weight, mutual information weight based on information entropy, etc.

[0072] The correlation between each model parameter and other model parameters can be calculated by a statistical method considering the interaction between the model parameters, which can be to collect the joint variation data of parameters and memory performance in multiple perturbation experiments or training trajectories, and to calculate the statistical association between each pair of parameters, so as to identify the parameter pairs with strong coupling relationship and provide input for interaction modeling.

[0073] The covariance and variance between the model parameters can be calculated based on the parameter gradient sequence or the perturbation response sequence, and the covariance matrix and its diagonal elements (variance) are estimated, so as to quantify the joint variation pattern of parameters and support the standardization calculation of interaction weight.

[0074] The standardized interaction weight can be obtained by dividing the covariance matrix element by the product of the corresponding parameter standard deviation to generate a correlation coefficient matrix as the interaction weight. Further, the operation can update the interaction weight dynamically by using a sliding time window to adapt to changes in the training stage, or apply a sparsification threshold to the interaction weight to reduce the computational overhead, so as to provide a structured representation of the inter-parameter coordination relationship for dynamic parameter optimization, and support a multi-parameter joint adjustment strategy.

[0075] For example, in the scenario of maintaining referential consistency in a long-context dialogue system, the method for improving the memory capacity of a large model can be: when processing a "how is the project he mentioned last week going?" type question in a multi-turn dialogue, the dynamic sensitivity analysis algorithm finds that the query weight parameter of a certain attention head has a high current partial derivative, but has deviated significantly from the initial reference value, and the non-linear influence factor greatly attenuates its effective sensitivity; at the same time, the parameter has a strong positive correlation (interaction weight 0.82) with another key projection parameter, and the optimizer adjusts both parameters simultaneously instead of updating them separately, thereby avoiding damaging the collaborative representation. Through iterative verification, the accuracy of the model in cross-turn entity reference is improved, and the total number of parameters is not increased.

[0076] The embodiment quantifies the importance of parameters by constructing the local sensitivity of component memory performance to model parameters and normalizing the ratio of the current value of the parameter to the memory performance, describes the accelerated effect of memory performance decline when the parameter deviates from the optimal reference value by introducing a non-linear influence factor based on an exponential decay function to overcome the distortion problem of traditional linear sensitivity analysis in non-convex high-dimensional space, and takes into account the parameter coordination effect in the optimization process instead of adjusting the parameters in isolation by modeling the coupling relationship between parameters through covariance and variance statistics and generating standardized interaction weights. The embodiment can accurately identify parameters and their interaction structures that play a key role in memory function, and achieve directional enhancement of memory-related subspaces without increasing the model capacity, thereby significantly improving the stability and generalization ability of large models in long-context understanding, knowledge consistency maintenance and other tasks, and endowing the entire memory enhancement process with interpretability, dynamic adaptability and closed-loop feedback characteristics, avoiding the blindness of experience-based parameter tuning and the high computational cost of structural modification.

[0077] In one embodiment, the calculation formula of the dynamic sensitivity factor is:

[0078]

[0079] wherein, the dynamic sensitivity factor of the jth model parameter of the ith component in the tth iteration is represented as the memory performance of the ith component in the tth iteration is represented as , denotes the initial memory performance of the i-th component; denotes the j-th model parameter value of the i-th component in the t-th iteration; denotes the partial derivative of the component memory performance with respect to the model parameter value; denotes the relative proportion of the model parameter to the component memory performance; denotes the non-linear influence factor; denotes the j-th model parameter reference value of the i-th component; denotes the decay coefficient; denotes the weighted sum of the model parameter interaction effects; denotes the interaction weight between the j-th and k-th model parameters.

[0080] wherein the dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration can be a quantitative index calculated in the t-th optimization iteration for the j-th model parameter within the i-th component, which comprehensively reflects the influence strength of the parameter on the memory performance, and can be used as a basis for adjusting the parameter in the dynamic parameter optimization algorithm, integrating the local sensitivity, non-linear decay and parameter interaction effects. In this embodiment, the dynamic sensitivity factor of the j-th model parameter of the i-th component in the t-th iteration can be calculated by integrating the partial derivative, the relative proportion, the non-linear influence factor and the interaction weighted sum. The memory performance of the i-th component in the t-th iteration can be a quantitative value of the memory capacity of the i-th component under the current parameter configuration at the t-th iteration, which can be used to update the performance benchmark in the sensitivity calculation in real time, supporting dynamic evaluation. For example, the memory performance of the i-th component in the t-th iteration can be recalculated by the monitoring module based on the new parameters performing tasks and counting memory error rates after each iteration.

[0081] The initial memory performance of the i-th component can be a memory capacity benchmark value of the component before optimization starts (t=0), which can be used as the initialization value of the memory performance in the t-th iteration and participate in the relative proportion calculation. In one exemplary embodiment, the initial memory performance of the i-th component can be obtained by the monitoring module for the first time evaluation in step S100. The j-th model parameter value of the i-th component in the t-th iteration can be the specific value of the j-th learnable parameter in the i-th component at the t-th iteration, which can be used as the current state input for sensitivity calculation to calculate the deviation and relative proportion.

[0082] The partial derivative of the component memory performance with respect to the model parameter value can be the first-order local rate of change of the component memory performance with respect to the specific model parameter, and can be used to characterize the direction and strength of the direct influence of a small change in the parameter on the memory performance. Further, the partial derivative of the component memory performance with respect to the model parameter value can be obtained by differentiating the curve fitted by the control variable experiment or approximated by using automatic differentiation technology. The relative proportion of the model parameter to the component memory performance can be the ratio of the current parameter value to the corresponding component memory performance, which is used to normalize the sensitivity dimension, and can be used to eliminate the sensitivity deviation caused by the absolute size of the parameter or the difference in the performance benchmark, and to improve the cross-component comparability.

[0083] The reference value of the jth model parameter of the ith component can be a benchmark configuration value used to measure whether the parameter deviates from the effective memory region, and can be used as the center point of the exponential decay in the nonlinear influence factor to determine the starting position of the decay. In one specific embodiment, the reference value of the jth model parameter of the ith component can include, but is not limited to, one or more of the parameter value at the initialization of the component, the snapshot value at the optimal memory performance on the historical validation set, the sliding average parameter trajectory, etc.

[0084] The decay coefficient can be a hyperparameter that controls the exponential decay rate in the nonlinear influence factor, and can be used to adjust the steepness of the sensitivity decline after the parameter deviates from the reference value, and to balance exploration and stability. Exemplarily, the decay coefficient can include, but is not limited to, a fixed decay coefficient, an adaptive decay coefficient (adjusted with the training stage), a task-dependent decay coefficient, etc. The weighted sum of the interaction effects of the model parameters can be the weighted sum of the interaction weights of the jth parameter with all other parameters k and the corresponding sensitivity or gradient terms, and can be used to quantify the comprehensive interaction effect of parameter j after considering the coupling relationship and to correct the isolated sensitivity evaluation. The interaction weight between the jth and kth model parameters can be the standardized statistical correlation strength between the parameters, which is used to weight the interaction effect, and can be used to reflect the degree of synergy or competition of the parameter pair (j, k) in the interaction weighted sum. Further, the interaction weight between the jth and kth model parameters can be the correlation coefficient obtained by dividing the covariance by the respective standard deviation, or a variant after sparsification.

[0085] The dynamic sensitivity factor of the jth model parameter of the ith component in the tth iteration can be a combination of the partial derivative, the relative proportion, the non-linear influence factor, and the interaction weighting and formula. Further, the dynamic sensitivity factor of the jth model parameter of the ith component in the tth iteration can be calculated by the above multi-source information fusion mechanism, so that a unified sensitivity index of fused multi-dimensional information can be generated to guide the differentiated parameter update. The memory performance of the ith component in the tth iteration can be that after the tth parameter update, the memory error rate of the component is re-evaluated on the verification task by the monitoring module and converted into a performance score. Further, the memory performance of the ith component in the tth iteration can be obtained by an online evaluation mechanism, so that real-time performance feedback required for dynamic sensitivity calculation can be provided. The initial memory performance of the component is initialized as the initial memory performance, which can be that when t = 0, the memory performance of the component is set to the initial memory performance calculated in step S100. In one specific embodiment, the initial memory performance of the component is initialized as the initial memory performance, which can be achieved by a pre-evaluation process, so that the sensitivity calculation can have an accurate benchmark at the optimization starting point.

[0086] The jth model parameter value of the ith component in the tth iteration can be read from the current model state. The jth model parameter value of the ith component in the tth iteration can be obtained by a model parameter cache mechanism, so that the current parameter state input can be provided for the sensitivity formula. The partial derivative of the memory performance of the component with respect to the model parameter value can be that the other parameters are fixed, the parameter value is perturbed , and the change is observed , and the derivative is obtained after fitting the local response curve. Further, the partial derivative of the memory performance of the component with respect to the model parameter value can be obtained by using the central difference method or fitting a surrogate model (such as local linear regression) and then analytically deriving, so that the local causal effect strength of the parameter on the memory performance can be obtained.

[0087] The relative proportion of the model parameter and the memory performance of the component can be a division operation. In one exemplary embodiment, the relative proportion of the model parameter and the memory performance of the component can be achieved by dimensionless processing, so that the dimensionless normalization of the sensitivity can be realized, and the cross-parameter comparability can be enhanced. The non-linear influence factor can be introduced to reflect the influence of the parameter deviating from the reference value, which can be calculated and included in the sensitivity formula as a multiplicative factor. Further, the non-linear influence factor can be introduced to reflect the influence of the parameter deviating from the reference value, which can be modeled by an exponential decay function, so that the parameter sensitivity in the area far from the reference value can be suppressed, and the actual contribution can be more truly reflected. The reference value of the jth model parameter of the ith component can be recorded before the optimization starts or selected from historical high-performance snapshots. Illustratively, the reference value of the jth model parameter of the ith component can be implemented through a snapshot storage mechanism, thereby providing an anchor point for nonlinear decay and defining the center of the “effective memory area”.

[0088] The decay coefficient configured to control the rate of nonlinear decay can be a preset value according to task complexity or experience, or a dynamically adjusted adaptive strategy. In a specific embodiment, the decay coefficient configured to control the rate of nonlinear decay can be implemented by fixing = 1.0 for general scenarios, or increasing with the training round to gradually tighten the optimization area, thereby adjusting the strictness of the nonlinear penalty and balancing robustness and convergence speed. The weighted sum of the interaction between model parameters can be implemented through a parameter coupling graph traversal mechanism, thereby explicitly introducing parameter coupling effects into sensitivity evaluation and avoiding isolated optimization. Determining the interaction weight between the jth and kth model parameters can be based on historical gradient or perturbation response sequence to calculate the covariance and variance, and normalize the correlation coefficient. Illustratively, determining the interaction weight between the jth and kth model parameters can be implemented by using the Pearson correlation coefficient as the interaction weight, or applying a threshold cut-off to the correlation coefficient to construct a sparse interaction graph, thereby quantifying the statistical dependence between parameter pairs and supporting the construction of interaction weighted sums.

[0089] For example, in the scenario of long-term reasoning tasks for legal documents, the method for increasing the memory capacity of the large model of the present embodiment can be that when answering a question such as “Does the defendant constitute a breach of contract according to Chapter 3, Article 5 and Chapter 7, Article 2?” the system finds that the bias parameter of a certain feedforward network layer has deviated significantly from the initialized reference value, and although its partial derivative is high, the nonlinear influence factor decays its sensitivity to 0.3; at the same time, this parameter has a strong negative interaction weight (-0.78) with another attention head weight parameter. After the dynamic sensitivity factor synthesizes this information, the optimizer only applies a small update to it, and synchronously adjusts the associated parameters to maintain collaborative representation. After three iterations, the accuracy of the model for joint reasoning of cross-chapter legal provisions is improved, and the model size is not increased.

[0090] The method for improving the memory capacity of a large model provided by the embodiment unifies the local sensitivity (partial derivative) of the component memory performance to a single parameter, the relative proportion of the current state of the parameter and the memory performance, the nonlinear decay effect caused by the deviation of the parameter from the optimal reference value (modeled by an exponential form of a nonlinear impact factor), and the coupling relationship between parameters (weighted summation through interaction weights) into a differentiable and optimizable index system through a dynamic sensitivity factor calculation formula, so that the dynamic sensitivity factor not only reflects the direct impact strength of the parameter on the memory, but also captures the marginal benefit change in a non-ideal state and the synergy / competition relationship with other parameters. The parameter adjustment guided thereby is no longer globally uniform or experience-driven, but rather precisely intervenes in the parameter subset with high sensitivity, high interaction, and low deviation, thereby efficiently improving the memory stability of each component without expanding the model structure, achieving fine-grained, dynamic, and closed-loop regulation of memory capacity, significantly alleviating context breakage, knowledge forgetting, and other problems, enhancing the coherence and generalization ability of the large model in long sequence tasks, while avoiding the computational redundancy and training instability caused by traditional methods due to coarse-grained parameter tuning or structural modification.

[0091] In one of the embodiments, the dynamic parameter optimization and memory capacity iterative improvement algorithm calculates the model parameter adjustment amount according to the dynamic sensitivity factor and the component memory performance, and reasonably determines the adjustment amplitude of the model parameters.

[0092] By calculating the unmemorability of the current component memory performance, multiplying the unmemorability by the dynamic sensitivity factor, and obtaining a preliminary adjustment amplitude as the numerator, the sum of the square of the dynamic sensitivity factor and a small positive number is introduced in the denominator; at the same time, a regulation coefficient is introduced to allow the step size of the adjustment to be controlled according to actual needs.

[0093] Among them, the unmemorability can be an index for quantifying the current memory failure degree of the component, and can be used as a driving factor for parameter adjustment strength, reflecting the urgency of memory enhancement of the current component. In an exemplary embodiment, the unmemorability can be calculated based on the component memory performance, and is usually 1 minus the memory performance or its monotonic increasing transformation. Further, the unmemorability can include but is not limited to one or more of context breakage, knowledge backtracking failure rate, long-range dependence loss index, etc.

[0094] The model parameter adjustment amount can be a numerical change amount of the model parameter applied in one optimization iteration, can be used to directly determine the direction and amplitude of parameter update, and can affect the improvement effect of component memory performance. In one specific embodiment, the model parameter adjustment amount can be calculated by the dynamic parameter optimization and memory amount iterative improvement algorithm according to the numerator, the denominator, and the adjustment coefficient combination. The adjustment amplitude can be the absolute value or relative scale of the model parameter adjustment amount, which can be used to control the degree of parameter change, and balance the optimization speed and stability. The square of the dynamic sensitivity factor can be a non-negative numerical value obtained by self-multiplication of the dynamic sensitivity factor, which can be used to suppress the excessive update of the high sensitivity parameter in the denominator, and enhance the numerical stability. The minimum positive number can be a constant close to zero but greater than zero, which can be used to ensure that the denominator is always positive, and guarantee the numerical robustness of the adjustment amount calculation. The numerator can be the dividend part in the parameter adjustment amount calculation formula, which is composed of the unmemorable degree and the dynamic sensitivity factor, and can be used to comprehensively reflect the severity of memory defects and the influence of parameters, and drive targeted parameter modification.

[0095] The denominator can be the divisor part in the parameter adjustment amount calculation formula, which is composed of the square of the dynamic sensitivity factor and the minimum positive number, and can be used to apply natural attenuation to the high sensitivity parameter, to avoid parameter oscillation caused by excessive sensitivity. The adjustment coefficient can be a user-configurable hyperparameter, which can be used to scale the parameter adjustment amount as a whole, and allow the optimization step to be flexibly controlled according to the task stage, data complexity, or convergence state. For example, the adjustment coefficient can adopt a large step size coefficient in the early stage of training, a small step size coefficient in the fine-tuning stage, and a dynamic scheduling coefficient, etc. The adjustment step size can be the final update amount actually applied to the model parameter, which is determined by the adjustment coefficient and the basic adjustment amplitude, and can be used to directly affect the convergence behavior of the optimization process and the efficiency of memory performance improvement.

[0096] According to the dynamic sensitivity factor and the component memory performance calculation model parameter adjustment amount, the numerator can be multiplied by the dynamic sensitivity factor, and the denominator can be divided by the square of the dynamic sensitivity factor plus the minimum positive number, and then multiplied by the adjustment coefficient. Further, the operation can be used for stable fine-tuning by fixing the adjustment coefficient (such as 0.1), or for dynamic convergence control by using an adjustment coefficient that decays with the training round (such as cosine annealing), so that a directional update amount matched with the degree of memory defect and parameter sensitivity is generated.

[0097] Reasonably determining the adjustment amplitude of the model parameter can be automatically scaling the adjustment amount through the numerator-denominator structure, and the parameters with high sensitivity but low memory defect obtain smaller updates, and the parameters with high defect and high sensitivity obtain moderate enhancement. Further, the operation can be achieved by the above-mentioned numerator-denominator structure, so that the training instability caused by blind large-scale update is avoided, and fine-grained parameter control is achieved.

[0098] The non-memorability of the current component memory performance can be obtained by performing a monotonic transformation (such as taking a complement, taking a negative logarithm, etc.) on the initial memory performance. Further, the operation can be implemented by non-memorability = 1-initial memory performance, or non-memorability = -log (initial memory performance + ε), so as to convert the memory performance into an error signal that needs to be corrected, and facilitate participation in optimization calculation.

[0099] The preliminary adjustment amplitude obtained by multiplying the non-memorability and the dynamic sensitivity factor can be a scalar multiplication operation that integrates the memory failure strength and the parameter influence. Further, the operation can be implemented by multiplying the numerical product of the aforementioned non-memorability and the dynamic sensitivity factor, so as to ensure that significant updates are triggered only when the component memory is poor and the parameter is sensitive. The sum of the square of the dynamic sensitivity factor and a small positive number in the denominator can be to construct the denominator term as (square of the dynamic sensitivity factor + small positive number). Further, the operation can be implemented by setting the small positive number to 1e-6 and the like, so as to automatically apply inhibition to high sensitivity parameters and prevent updates from being too large to cause oscillation. Introducing an adjustment coefficient to control the step size of the adjustment can be to multiply the aforementioned calculation result by a configurable scalar coefficient. Further, the operation can be implemented by fixing the adjustment coefficient (such as 0.1) for stable fine-tuning, or an adjustment coefficient that decays with training rounds (such as cosine annealing) for dynamic convergence control, so as to provide an external intervention interface to adapt to different training stages or task requirements.

[0100] For example, in the scenario of role consistency maintenance in a multi-round dialogue system, the method for improving the memory capacity of a large model of the present embodiment can be: when processing more than ten rounds of dialogue between a user and a virtual role, the monitoring module finds that a certain feedforward network layer frequently forgets the initial settings (such as occupation, personality) of the role after the 7th round; dynamic sensitivity analysis shows that the bias parameter of this layer is highly sensitive to such memory; the algorithm calculates its non-memorability as 0.65, the dynamic sensitivity factor as 0.8, the numerator of the formula as 0.52, the denominator as 0.64 + 1e-6 ≈ 0.64, and the basic adjustment amplitude as ≈ 0.8125; if the adjustment coefficient is set to 0.2, the final adjustment step is 0.1625, and the bias parameter is fine-tuned accurately; subsequent verification shows that the role information retention rate is improved, and no other component performance is degraded.

[0101] The method for improving the memory capacity of a large model provided by the embodiment adjusts the amplitude of model parameter adjustment by calculating the model parameter adjustment amount based on the dynamic sensitivity factor and the component memory performance calculation model, and reasonably determines the adjustment amplitude of the model parameter. The non-memorability of the current component memory performance is calculated, the non-memorability is multiplied by the dynamic sensitivity factor to obtain a preliminary adjustment amplitude as the numerator, and the square of the dynamic sensitivity factor and a small positive number are introduced into the denominator. At the same time, an adjustment coefficient is introduced to control the step length of the adjustment. By taking the non-memorability as a quantitative driving signal of memory failure, combining the dynamic sensitivity factor to construct the numerator to reflect the correction demand intensity, using the square of the dynamic sensitivity factor and the small positive number to form the denominator to suppress the oscillation of high sensitivity parameters, and flexibly regulating the final update step length with the help of the adjustment coefficient, the technical effects of precise and robust optimization of key memory-related parameters without changing the model structure, effective balance of convergence speed and stability, significant improvement of memory continuity and generalization ability of the large model in long context, multi-round dialogue or knowledge-intensive tasks, and maintenance of low computational overhead can be achieved.

[0102] In one embodiment, the calculation formula of the model parameter adjustment amount is as follows:

[0103]

[0104] wherein, represents the jth model parameter adjustment amount of the ith component in the tth iteration; represents the adjustment coefficient; represents the non-memorability of the memory performance of the ith component in the tth iteration; represents the memory performance of the ith component in the tth iteration; represents the normalized denominator; represents a small positive number; represents the dynamic sensitivity factor of the jth model parameter of the ith component in the tth iteration.

[0105] The jth model parameter adjustment amount of the ith component in the tth iteration can be a numerical update amount calculated for the jth specific parameter in the ith component of the large model in the tth optimization iteration, which can be used as a direct basis for parameter updating to achieve fine-grained and time-sequenced regulation of memory-related parameters. In one exemplary embodiment, the jth model parameter adjustment amount of the ith component in the tth iteration can include but is not limited to one or more of attention head weight adjustment amount, feedforward network bias adjustment amount, layer normalization scaling factor adjustment amount, etc.

[0106] The unmemorability of the memory performance of the ith component can be a quantitative index representing the degree of memory failure derived from the current memory performance of the ith component, which can be used to generate the parameter adjustment amount, ensuring that the update strength is positively correlated with the severity of the memory problem. Further, the unmemorability of the memory performance of the ith component can be calculated based on the memory performance of the ith component, and a monotonic transformation (such as taking the complement, negative logarithm, etc.) is performed on the memory performance value to obtain the unmemorability, thereby converting the performance index into an error driving signal, adapting to the input requirements of the optimization mechanism. The memory performance of the ith component at the tth iteration can be the memory capability evaluation value obtained by the ith component based on the monitoring module at the tth iteration, which can be used as the basis for calculating the unmemorability, reflecting the memory state of the component at the current training / inference stage.

[0107] The normalization denominator can be a denominator item in the parameter adjustment amount formula, which consists of the square of the dynamic sensitivity factor and a minimum positive number, and can be used to adaptively scale the adjustment amount, suppress excessive update of high sensitivity parameters, and improve numerical stability. In one specific embodiment, the normalization denominator can include but is not limited to a high sensitivity suppression denominator, a low sensitivity reservation denominator, a numerical stability denominator, etc. Further, the normalization denominator is constructed to stabilize the parameter adjustment process, and the square of the dynamic sensitivity factor and a minimum positive number are used to constitute the denominator item, which automatically attenuates the high sensitivity parameters and prevents parameter oscillation or divergence caused by high sensitivity. In this embodiment, the minimum positive number is introduced in the normalization denominator to prevent division by zero or numerical instability, and a constant close to zero but strictly greater than zero (such as 1e-8) is added in the denominator expression to ensure the mathematical legality of the division operation and the robustness of the numerical calculation.

[0108] The dynamic sensitivity factor of the jth model parameter of the ith component in the tth iteration can be a time-varying sensitivity measure of the degree of influence of the jth parameter in the ith component on the memory performance in the tth iteration, which can be used to identify the importance of the parameter to the memory function in the current context and determine the weight of the parameter in the adjustment. The adjustment amount of the jth model parameter of the ith component in the tth iteration can be calculated by multiplying the adjustment coefficient by (unmemorability x dynamic sensitivity factor) and then dividing by (dynamic sensitivity factor2+ minimum positive number). Further, the adjustment amount of the jth model parameter of the ith component in the tth iteration can be calculated by combining the adjustment coefficient with unmemorability and dynamic sensitivity factor for parameter updating, integrating the three according to the formula structure to form a directional and weighted adjustment amount, thereby generating a parameter updating signal with memory orientation, component awareness and numerical robustness. For example, combining the adjustment coefficient with unmemorability and dynamic sensitivity factor for parameter updating can independently perform the update outside backpropagation, without relying on loss gradient, or input the adjustment amount as a gradient replacement item into a standard optimizer (such as Adam), so that the memory enhancement goal is decoupled from the task loss optimization, and the function-oriented parameter evolution is realized.

[0109] For example, in the scenario of legal text cross-chapter reference understanding task, the method for improving the memory amount of the large model of the embodiment can be that when processing a question and answer containing dozens of mutually referenced legal provisions, the attention head of a certain Transformer layer is identified by the monitoring module in the 5th iteration that the memory error rate of the principle of "superior law first" is 40%; the algorithm calculates its unmemorability as 0.6 and the dynamic sensitivity factor as 0.75; substituting the formula, assuming that the adjustment coefficient is 0.15 and the minimum positive number is 1e-8, then the adjustment amount = 0.15 x (0.6 x 0.75) / (0.75²+1e-8) ≈ 0.12; the system applies this update to the query weight parameter of the attention head; subsequent verification shows that its accuracy in processing "how to apply this law and related laws in conflict" type problems is significantly improved, and other semantic understanding abilities are not affected.

[0110] The method for improving the memory capacity of a large model provided by the embodiment calculates the model parameter adjustment amount by multiplying the adjustment coefficient by the product of the unmemorability and the dynamic sensitivity factor, and dividing the sum of the square of the dynamic sensitivity factor and the minimum positive number, the unmemorability reflects the severity of the current memory failure of the component, the dynamic sensitivity factor depicts the influence weight of a specific parameter on the memory capacity, and the product of the two constitutes the basis for the adjustment direction and intensity, the normalized denominator realizes adaptive scaling by the square of the dynamic sensitivity factor plus the minimum positive number to suppress the excessive update of high sensitivity parameters and ensure that low sensitivity parameters can still be effectively optimized, the adjustment coefficient provides global step length control to adapt to different training stages or task requirements, and the parameter update can have fine granularity, component awareness and memory-oriented characteristics, breaking through the limitations of traditional optimizers that only rely on loss gradients and ignore memory function characteristics, precisely strengthening the optimization of key memory parameters without increasing the model parameter amount or introducing external structures, significantly improving the stability and generalization ability of large models in long context reasoning, knowledge consistency maintenance and other scenarios, while maintaining training efficiency and numerical robustness, achieving efficient, interpretable and closed-loop memory enhancement effect.

[0111] In one of the embodiments, the dynamic parameter optimization and memory capacity iterative improvement algorithm updates the model parameter value based on the current model parameter value and the model parameter adjustment amount, to ensure that the adjusted model parameter is computationally feasible, a boundary constraint is introduced, the sum of the current model parameter value and the model parameter adjustment amount is compared with the maximum allowed value of the model parameter, the smaller value of the two is taken, and then compared with the minimum allowed value of the model parameter, the larger value of the two is taken as the final updated model parameter value, to ensure that the adjusted parameter will not exceed the limit of hardware resources or model architecture;

[0112] The current model parameter value can be the actual value of a parameter in the model before the start of this iteration, which can be used as the starting point for parameter update and participate in adjustment amount superposition operation. The model parameter adjustment amount can be the recommended change amount output by the external optimizer, which is added to the current model parameter value to generate a candidate new parameter value without constraint. In the embodiment, the model parameter value can be updated based on the current model parameter value and the model parameter adjustment amount, which can be implemented by performing addition operation: the temporary parameter is equal to the sum of the current model parameter value and the model parameter adjustment amount. Further, the operation can be realized by standard gradient descent, adaptive learning rate method or other first-order optimization strategy, so as to generate a candidate new parameter value reflecting the optimization direction.

[0113] The boundary constraint can be an upper and lower limit restriction mechanism imposed on the value range of the model parameter after updating, which can be used to prevent the parameter from exceeding the numerical range allowed by the hardware representation capability or the model architecture design due to excessive adjustment, and to ensure the stability of training and the feasibility of deployment. In an exemplary embodiment, the boundary constraint can be performed by presetting the maximum and minimum allowed values of the model parameters, and performing a clipping operation after parameter updating. For example, the boundary constraint can include one or more of the numerical boundary based on FP16 precision, the parameter boundary based on the input domain of the activation function, the statistical boundary based on the weight initialization distribution, etc.

[0114] The maximum allowed value of the model parameter can be an upper limit of the parameter determined by the hardware precision, the saturation region of the activation function, or the requirement for training stability, which can be used as an upper bound of the boundary constraint to prevent the parameter from being too large to cause gradient explosion or numerical overflow. The minimum allowed value of the model parameter can be a lower limit of the parameter determined by the hardware precision or the model structure requirement, which can be used as a lower bound of the boundary constraint to prevent the parameter from being too small to cause information loss or underflow. In a specific embodiment, the boundary constraint introduces the sum of the current model parameter value and the model parameter adjustment amount, and compares it with the maximum allowed value of the model parameter to take the smaller value, which can be an upper bound clipping operation on the temporary parameter: the temporary parameter is equal to the smaller one of the sum and the maximum allowed value. Further, this operation can be implemented by element-by-element comparison and assignment, so that the parameter can be prevented from exceeding the feasible upper limit due to excessive forward updating.

[0115] The result of the previous step is compared with the minimum allowed value of the model parameter, and the larger value is taken as the final updated model parameter value, which can be a lower bound clipping operation on the result of the previous step: the updated model parameter value is equal to the larger one of the result and the minimum allowed value. Further, this operation can be implemented by conditional judgment or vectorized comparison, so that the parameter can be ensured to be always within the legal numerical interval. The updated model parameter value can be the final parameter value after boundary constraint processing, which can be used for subsequent forward propagation and performance evaluation. In an exemplary embodiment, the updated model parameter value can be obtained by first calculating the sum of the current value and the adjustment amount, and then comparing and clipping the maximum and minimum allowed values in turn. The limitation of hardware resources or model architecture can be an external or structural constraint condition that affects the feasibility of the model parameter value, which can be used to provide a basis for setting the boundary constraint to ensure that the parameter update result can be deployed and executed on the target platform. Ensuring that the adjusted parameter does not exceed the limitation of hardware resources or model architecture can be achieved by the above-mentioned double-sided clipping operation to comply with the parameter value domain, so as to improve the numerical stability and deployment compatibility of the model in the training and inference stages.

[0116] Further updating the memory performance of the component based on the updated model parameter value;

[0117] The further updating of the memory performance of the component based on the updated model parameter value can be performing forward inference using the new parameters, reevaluating the number of memory errors under the assistance of the monitoring module, and calculating a new memory performance. Further, the operation can be implemented by running an inference task on the validation subset and counting the key fact backtracking accuracy, so as to establish a parameter-performance mapping feedback to support the next round of optimization.

[0118] The adjusted model parameter directly reflects the change in the memory performance of the component, and the actual model parameter change in the current iteration is calculated, that is, the updated model parameter value minus the adjusted model parameter value, divided by the reference value of the model parameter, to obtain a dimensionless relative change amplitude, which is multiplied by the corresponding dynamic sensitivity factor. The unmemorability is introduced to gradually reduce the component memory performance improvement amplitude when it approaches the theoretical upper limit.

[0119] The adjusted model parameter value can be the same as the current model parameter value, indicating the original value before the parameter update operation occurs, and can be used to calculate the actual change, which is used as the benchmark of the relative change amplitude. The reference value of the model parameter can be a scale benchmark for normalizing the parameter change, which is usually the absolute value, the initial value or the moving average value of the parameter, and can be used to eliminate the dimensional difference of different parameters, so that the relative change amplitude has cross-parameter comparability. In one specific embodiment, the actual model parameter change in the current iteration (updated minus adjusted) can be the updated model parameter value minus the adjusted model parameter value. Further, the operation can be implemented by parameter-by-parameter difference operation, so that the real effective parameter change can be reflected, which can be smaller than the original adjustment amount due to boundary constraints.

[0120] The actual model parameter change divided by the reference value of the model parameter to obtain a dimensionless relative change amplitude can be a standardized update strength indicator obtained by dividing the actual parameter change by the reference value. Further, the operation can be implemented by taking the initial absolute value of the parameter as the reference value or taking the exponential moving average of the absolute value of the parameter in the sliding window as the reference value, so that the update strength standardization across the parameter scale can be achieved. The dimensionless relative change amplitude can be a measure of the relative significance of parameter update, which can be used to couple with the dynamic sensitivity factor to evaluate the optimization effectiveness. Multiplying the relative change amplitude by the corresponding dynamic sensitivity factor can be to calculate the coupling index equal to the relative change amplitude multiplied by the dynamic sensitivity factor. Further, the operation can be implemented by looking up the table or online estimating the dynamic sensitivity factor, so that the effectiveness and influence of the parameter update can be comprehensively measured to guide the subsequent iteration strategy.

[0121] The theoretical upper limit can be the limit of the memory performance of the component under a specific task or data distribution, which can be used as a reference point for the non-memorability decay mechanism, and the optimization strength is automatically reduced when the memory performance approaches the value. The introduction of non-memorability makes the memory performance of the component gradually decrease when approaching the theoretical upper limit. In subsequent iterations, the non-memorability is used as a decay factor for the adjustment amount or the adjustment coefficient, and the value tends to zero as the memory performance approaches the theoretical upper limit. Further, the operation can be implemented by setting the non-memorability equal to one minus the ratio of the current memory performance to the theoretical upper limit, that is, Or the non-memorability is equal to a function decaying in a negative exponential form, so as to form an adaptive convergence mechanism to avoid invalid optimization or overfitting in the performance saturation area.

[0122] For example, in the scenario of fact consistency maintenance in a knowledge-intensive question answering system, the method for improving the memory capacity of a large model can be: when answering a question involving multi-hop reasoning, a certain attention head causes the softmax output to saturate due to excessively large parameters, frequently ignoring early facts; the algorithm calculates its adjustment amount as +0.8, but the maximum allowed value of the parameter is 1.5, and the current value is 1.45, so the updated value is truncated to 1.5; then run the verification task with the new parameter, and the memory performance increases from 0.62 to 0.68; the relative change amplitude is (1.5-1.45) / 1.0=0.05, multiplied by the sensitivity factor 0.9 to get 0.045; at the same time, since the current memory performance is still 0.85 away from the theoretical upper limit, the non-memorability is 0.37, allowing further optimization; if the subsequent performance reaches 0.82, the non-memorability decreases to 0.035, greatly inhibiting further adjustment, and achieving smooth convergence.

[0123] The method for improving the memory capacity of a large model provided in this embodiment updates the model parameter value based on the current model parameter value and the model parameter adjustment amount, introduces boundary constraints to truncate the updated result, further updates the memory performance of the component based on the updated model parameter value, calculates the actual model parameter change amount and normalizes it to a dimensionless relative change amplitude, multiplies it by a dynamic sensitivity factor, and introduces non-memorability as a decay mechanism when the memory performance approaches the theoretical upper limit. By embedding boundary constraints determined by hardware resources or model architecture limitations in parameter updating to ensure numerical feasibility, by normalizing parameter changes using reference values and coupling with dynamic sensitivity factors to evaluate optimization effects across parameter scales, and by automatically decaying adjustment strength as the memory performance approaches the theoretical upper limit to form adaptive convergence, the method can efficiently and safely improve the memory performance of a large model in key tasks such as long sequence dependence, context coherence, and knowledge backtracking without expanding the model structure, significantly outperforming traditional experience-based parameter tuning or coarse-grained optimization strategies.

[0124] In one of the embodiments, the dynamic parameter optimization and memory quantity iterative promotion algorithm calculates the memory quantity of the entire large model based on the component memory performance after iteration, and evaluates the error between the memory quantity of the entire large model and the target value. If the error is positive, it indicates that the current memory quantity of the entire large model has not reached the target, that is, the current model parameter adjustment is insufficient, and iteration needs to be continued to improve. If the error is zero or negative, it indicates that the target has been reached or exceeded, and the iteration process is stopped.

[0125] The component memory performance after iteration can be a memory performance index of each component obtained after re-evaluation after one round of dynamic parameter optimization. In this embodiment, the component memory performance after iteration can be used as an input basis for calculating the current entire machine memory quantity, reflecting the actual influence of this round of parameter adjustment on the component memory capability.

[0126] The memory quantity of the entire large model can be a quantitative index representing the overall memory capability of the current model, which is aggregated from the component memory performance after iteration. For example, the memory quantity of the entire large model can follow the aggregation rule (such as weighted average) of the initial memory quantity, but the input is the component performance after the latest round of optimization. Further, the memory quantity of the entire large model can be used to compare with the target value to generate an error signal to drive the iteration control logic.

[0127] The target value can be a preset expected threshold of the overall memory quantity of the large model. In one exemplary embodiment, the target value can be used as a convergence criterion for the optimization process, providing a clear memory enhancement target orientation. The error can be the difference between the memory quantity of the entire large model and the target value (target value minus current memory quantity). In one specific embodiment, the error can be calculated by numerical subtraction, and the sign determines whether to continue iteration. Further, the error can be used as a basis for judging whether to terminate or continue iteration, realizing closed-loop feedback control.

[0128] The memory quantity of the entire large model can be calculated based on the component memory performance after iteration, that is, after each round of parameter optimization, the same aggregation rule as the initial memory quantity is called to fuse the updated component memory performance into the global memory quantity. Further, this operation can be realized through a unified aggregation function, so that the real-time quantification of the current model memory capability can be realized, providing a basis for error calculation. Evaluating the error between the memory quantity of the entire large model and the target value can be performing an arithmetic operation of the target value minus the current memory quantity of the entire large model to obtain a signed error value. Further, this operation can be realized through a standard numerical comparison module, so that an interpretable optimization deviation signal can be generated, clearly indicating the gap direction and existence of the current state and the target.

[0129] The judgment of whether the error is a positive number to determine whether to continue iteration can be checking the sign of the error value: if greater than zero, triggering the next round of sensitivity analysis and parameter optimization; otherwise, terminating the process. For example, the judgment logic can be implemented through a conditional branch control structure, so that an adaptive termination mechanism based on the target achievement degree can be established to avoid invalid iterations. Stopping the iteration process when the error is zero or negative can be interrupting the execution loop of the dynamic parameter optimization and memory iteration improvement algorithm when the error is ≤0. Further, the operation can be implemented by the flow controller in response to the error signal, so that over-adjustment and waste of computing resources can be prevented, and the optimization process can be ensured to converge in time after meeting the target.

[0130] For example, in the scenario of legal text cross-paragraph reasoning tasks, the method for improving the memory capacity of a large model according to the embodiment can be: in processing a question and answer task that needs to refer to multiple legal provisions for comprehensive judgment, the system presets the target memory capacity as 0.92. After the first round of optimization, the memory capacity of the whole machine is 0.85, and the error is positive, triggering the second round of sensitivity analysis, which finds that a certain attention head is insufficient in long-term dependence modeling of legal provision numbers; after targeted fine-tuning, the memory capacity increases to 0.93, and the error is -0.01, and the system automatically terminates iteration. The final model significantly improves the consistency and accuracy of answers to questions such as “whether Z behavior is constituted according to relevant legal provisions and other legal provisions” without increasing the number of parameters.

[0131] The method for improving the memory capacity of a large model provided by the embodiment calculates the memory capacity of the whole large model based on the memory performance of the components after iteration, evaluates the error between the memory capacity of the whole large model and the target value, continues iteration if the error is a positive number, and stops the iteration process if the error is zero or negative. By aggregating the memory performance of the components into the memory capacity index of the whole machine and quantitatively comparing it with the preset target to generate an error signal, and triggering a new round of parameter optimization or terminating the process according to the sign of the error, a closed-loop feedback mechanism based on the target value can be introduced, the parameter adjustment process has a clear convergence criterion, over-adjustment or under-adjustment is avoided, the context retention capability and knowledge backtracking stability in long sequence modeling are accurately improved without expanding the model structure, and the technical effect of significantly enhancing the generalization performance of the large model in complex reasoning tasks is achieved.

[0132] In addition, to achieve the above-mentioned purpose, the application further provides a system for improving the memory capacity of a large model, which comprises:

[0133] The memory monitoring module is used for deploying a monitoring module in the model training or reasoning process, recording the number of memory errors and the total number of task processing of each component of the large model in the task processing stage, and calculating the initial memory performance of each component according to the number of memory errors and the total number of task processing;

[0134] The performance evaluation module is configured to calculate an initial memory amount of the whole large model based on the initial memory performance of each component, as a starting point of dynamic parameter optimization.

[0135] The sensitivity analysis module is configured to analyze the relationship between the component memory performance and the model parameters by a dynamic sensitivity analysis algorithm, and calculate a dynamic sensitivity factor of each component model parameter.

[0136] The parameter optimization module is configured to dynamically adjust the model parameters by a dynamic parameter optimization and memory amount iterative improvement algorithm based on the dynamic sensitivity factor, the component memory performance and the model parameters, and gradually improve the memory amount of the components and the whole machine by combining the sensitivity analysis and iterative verification.

[0137] Other embodiments or specific implementations of the system for improving the memory amount of a large model according to the present application can refer to the above-mentioned method embodiments, and will not be described here.

[0138] In addition, to achieve the above-mentioned purpose, the present application also provides an apparatus for improving the memory amount of a large model, which comprises a memory, a processor and a program for improving the memory amount of a large model stored in the memory and executable on the processor, and the program for improving the memory amount of a large model is configured to implement the steps of the method for improving the memory amount of a large model according to any one of the above-mentioned embodiments.

[0139] In addition, to achieve the above-mentioned purpose, the present application also provides a medium having a program for improving the memory amount of a large model stored thereon, and the program for improving the memory amount of a large model is executed by a processor to implement the steps of the method for improving the memory amount of a large model according to any one of the above-mentioned embodiments.

[0140] The above-mentioned are only the preferred embodiments of the present application, and do not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation according to the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for improving the memory capacity of a large model, characterized by, The method comprises: deploying a monitoring module in the model training or inference process, recording the number of memory errors and the total number of task processing of each component of the large model in the long document question and answer task processing stage, and calculating the initial memory performance of each component according to the number of memory errors and the total number of task processing; based on the initial memory performance of each component, the initial memory amount of the entire large model is calculated as the starting point of dynamic parameter optimization; through a dynamic sensitivity analysis algorithm, the relationship between the component memory performance and the model parameters is analyzed, and the dynamic sensitivity factor of each component model parameter is calculated; based on the dynamic sensitivity factor, the component memory performance and the model parameters, the model parameters are dynamically adjusted through the dynamic parameter optimization and memory amount iterative improvement algorithm, and combined with sensitivity analysis and iterative verification, the memory amount of the component and the whole machine in the long document question and answer task processing stage is gradually improved; wherein the dynamic sensitivity analysis algorithm evaluates the direct influence of model parameter changes on component memory performance, fits a curve by controlling variable experiments, and obtains the partial derivative of component memory performance with respect to model parameters; the partial derivative is multiplied by the ratio of component model parameter value to component memory performance; a nonlinear influence factor is introduced again, the influence when the model parameters deviate from the reference value is described through an exponential decay function, and the nonlinear effect of the model parameters deviating from the optimal state on the component memory performance in the actual training is simulated; considering the interaction between model parameters, the correlation between each model parameter and other model parameters is calculated by statistical method, the covariance and variance between model parameters are calculated, and a standardized interaction weight is obtained; wherein the calculation formula of the dynamic sensitivity factor is: ； wherein, denotes the dynamic sensitivity factor of the jth model parameter of the ith component in the tth iteration; denotes the memory performance of the ith component in the tth iteration, initially , denotes the initial memory performance of the ith component; denotes the value of the jth model parameter of the ith component in the tth iteration; denotes the partial derivative of the memory performance of the component with respect to the value of the model parameter; denotes the relative proportion of the model parameter to the memory performance of the component; denotes the non-linear influence factor; denotes the reference value of the jth model parameter of the ith component; denotes the decay coefficient; denotes the weighted sum of the interaction of the model parameters; denotes the interaction weight between the jth and kth model parameters.​ 2. The method of increasing the amount of memory of a large model of claim 1, wherein, the dynamic parameter optimization and memory amount iterative improvement algorithm calculates the model parameter adjustment amount according to the dynamic sensitivity factor and the component memory performance, and reasonably determines the adjustment amplitude of the model parameters; by calculating the unmemorability of the current component memory performance, the unmemorability is multiplied by the dynamic sensitivity factor to obtain a preliminary adjustment amplitude as the numerator, and the square of the dynamic sensitivity factor and a small positive number are introduced in the denominator; at the same time, a regulation coefficient is introduced to allow the step length of adjustment to be controlled according to actual needs. 3.The method of claim 2, wherein, The calculation formula of the model parameter adjustment amount is as follows: ； wherein, represents an adjustment amount of the jth model parameter of the ith component at the tth iteration; represents an adjustment coefficient; represents the non-memorability of the memory performance of the ith component at the tth iteration; represents the memory performance of the ith component at the tth iteration; represents a normalization denominator; represents a very small positive number; represents a dynamic sensitivity factor of the jth model parameter of the ith component at the tth iteration. 4.The method of claim 2, wherein, the dynamic parameter optimization and memory amount iterative improvement algorithm updates the model parameter value based on the current model parameter value and the model parameter adjustment amount, in order to ensure that the adjusted model parameter is computationally feasible, a boundary constraint is introduced, the sum of the current model parameter value and the model parameter adjustment amount is compared with the maximum allowed value of the model parameter, the smaller value of the two is taken, and then compared with the minimum allowed value of the model parameter, the larger value of the two is taken as the final updated model parameter value, to ensure that the adjusted parameter will not exceed the limit of hardware resources or model architecture; based on the updated model parameter value, the memory performance of the component is further updated; The change of the actual model parameter in this iteration is calculated directly by the adjusted model parameter, that is, the updated model parameter value minus the adjusted model parameter value, divided by the reference value of the model parameter, to obtain a dimensionless relative change amplitude, which is multiplied by the corresponding dynamic sensitivity factor; The non-memorability is introduced, so that the memory performance improvement amplitude of the component gradually decreases when it approaches the theoretical upper limit.

5. The method of increasing the amount of memory of a large model of claim 1, wherein, The dynamic parameter optimization and memory quantity iterative improvement algorithm calculates the memory quantity of the entire large model based on the memory performance of the component after iteration, and evaluates the error between the memory quantity of the entire large model and the target value. If the error is positive, it means that the memory quantity of the entire large model has not reached the target, that is, the effect of the current model parameter adjustment is insufficient, and iteration needs to be continued to improve. If the error is zero or negative, it means that the target has been reached or exceeded, and the iteration process is stopped.

6. A system for improving the memory capacity of a large model, characterized by, The system comprises: A memory monitoring module is configured to deploy a monitoring module in the model training or inference process, record the number of memory errors and the total number of task processing of each component of the large model in the long document question and answer task processing stage, and calculate the initial memory performance of each component based on the number of memory errors and the total number of task processing; A performance evaluation module is configured to calculate the initial memory quantity of the entire large model based on the initial memory performance of each component, as the starting point of dynamic parameter optimization; A sensitivity analysis module is configured to analyze the relationship between the memory performance of the component and the model parameter by a dynamic sensitivity analysis algorithm, and calculate the dynamic sensitivity factor of each component model parameter; A parameter optimization module is configured to dynamically adjust the model parameter based on the dynamic sensitivity factor, the memory performance of the component and the model parameter by the dynamic parameter optimization and memory quantity iterative improvement algorithm, and gradually improve the memory quantity of the component and the entire machine in the long document question and answer task processing stage through sensitivity analysis and iterative verification. The dynamic sensitivity analysis algorithm evaluates the direct influence of the change of the model parameter on the memory performance of the component, fits a curve by a control variable experiment, obtains the partial derivative of the memory performance of the component with respect to the model parameter, and multiplies the partial derivative by the ratio of the model parameter value of the component to the memory performance of the component. A nonlinear influence factor is introduced again, which describes the influence of the deviation of the model parameter from the reference value through an exponential decay function, and simulates the nonlinear effect of the deviation of the model parameter from the optimal state on the memory performance of the component in the actual training. The interaction between the model parameters is considered, and the correlation between each model parameter and other model parameters is calculated by a statistical method, the covariance and variance between the model parameters are calculated, and a standardized interaction weight is obtained. The calculation formula of the dynamic sensitivity factor is as follows: ； wherein, denotes the dynamic sensitivity factor of the jth model parameter of the ith component in the tth iteration; denotes the memory performance of the ith component in the tth iteration, initially , denotes the initial memory performance of the ith component; denotes the jth model parameter value of the ith component in the tth iteration; denotes the partial derivative of the memory performance of the component with respect to the model parameter value; denotes the relative proportion of the model parameter to the memory performance of the component; denotes the non-linear influence factor; denotes the jth model parameter reference value of the ith component; denotes the decay coefficient; denotes the weighted sum of the model parameter interaction; denotes the interaction weight between the jth and kth model parameters.​ 7. A device for improving the memory capacity of a large model, characterized by, The device comprises a memory, a processor, and a program for improving the memory quantity of a large model stored on the memory and executable on the processor, which is configured to implement the steps of the method for improving the memory quantity of a large model according to any one of claims 1 to 5. The device comprises a memory, a processor, and a program for improving the memory quantity of a large model stored on the memory and executable on the processor, which is configured to implement the steps of the method for improving the memory quantity of a large model according to any one of claims 1 to 5.

8. A medium characterized by, The medium stores a program for improving the memory capacity of a large model, and the program for improving the memory capacity of a large model, when executed by a processor, implements the steps of the method for improving the memory capacity of a large model according to any one of claims 1 to 5.

Images