A large language model test-time learning method, device, equipment and medium

Abstract

The application discloses a large language model test learning method, device, equipment and medium, test data is input into a large language model with low rank adaptive parameters, the prediction score of each token in the vocabulary at each generation step is obtained; the token-level evidence quality of the generation step is extracted based on the prediction score, and the token-level cognitive uncertainty of the generation step is calculated; the token-level cognitive uncertainty is smoothed, the stable area of the generation step is determined according to the smoothed uncertainty, and the mask of the model update window is constructed; the mask entropy loss is calculated based on the mask, and the low rank adaptive parameters are updated through back propagation. The application can solve the problem of adaptive drift in model updating in related technologies.

Description

Technical Field

[0001] This application relates to the field of large language model technology, and in particular to a learning method, apparatus, device and medium for testing large language models. Background Technology

[0002] Large Language Models (LLMs) often face the problem of inconsistent training and testing data distributions in real-world deployments. Specifically, this manifests as testing data containing unfamiliar domain terminology (e.g., from vertical fields like medicine, finance, and agriculture), diverse language variants (e.g., dialects and slang), or dynamically changing user intent. This distribution shift significantly degrades the generation quality and task performance of large language models, severely limiting their reliability in complex real-world environments.

[0003] To address the distribution offset problem, Test-Time Learning (TTL) techniques emerged. The most representative approach is based on entropy minimization.

[0004] However, existing test-time learning methods based on entropy minimization still have the following drawbacks in practical applications: Inaccurate uncertainty estimation poses a risk of "blind confidence": Existing techniques typically measure uncertainty solely by calculating entropy values ​​based on the predicted probabilities of the model output. However, a high predicted probability does not necessarily equate to high-quality evidence supporting the model. The model may exhibit "blind confidence" by displaying a high probability even in the absence of sufficient evidence. Existing methods often neglect the evidentiary quality of the Top-K candidate terms in the Logits distribution, leading to biased uncertainty estimation and consequently misleading the model's update direction.

[0005] The update process is unstable and susceptible to noise interference: Existing test-time learning typically updates parameters directly at each generation step or for each batch of data. However, the uncertainty of the model often fluctuates during the generation process. If an update is forced at a step where the model is hesitant or the prediction is unstable, noisy gradients will be introduced, leading to error accumulation and even further deterioration of model performance (i.e., "catastrophic forgetting" or "adaptive drift").

[0006] The loss function design is simplistic and lacks consideration for the quality of evidence: Existing loss functions typically treat the entropy loss of all generation steps equally, failing to weight it according to the reliability of evidence at each step. Furthermore, existing methods lack constraints on the changing trends of uncertainty; drastic jumps in uncertainty between adjacent generation steps can affect the smoothness and robustness of the model's adaptation process.

[0007] The trade-off between computational efficiency and adaptive performance: Some adaptive methods involve updating all parameters, resulting in high computational costs and making them difficult to deploy in real-world scenarios. Existing lightweight methods often compromise on adaptive performance, struggling to achieve accurate evidence-guided updates while maintaining high efficiency.

[0008] Therefore, how to provide a test-time learning method that can more accurately estimate cognitive uncertainty, effectively screen stable update regions, and combine evidence quality weighted loss to improve the adaptability and robustness of large language models in distribution shift scenarios has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] This application provides a learning method, apparatus, device, and medium for testing large language models to solve the problem of adaptive drift in model updates in related technologies.

[0010] In a first aspect, embodiments of this application provide a large language model test-time learning method, the large language model test-time learning method comprising: Input the test data into a large language model with low-rank adaptation parameters and obtain the prediction score of the word in the vocabulary at each generation step; Based on the predicted score, extract the lexical-level evidence quality of the generation step and calculate the lexical-level cognitive uncertainty of the generation step. The cognitive uncertainty at the word level is smoothed, the stable region of the generation step is determined based on the smoothed uncertainty, and a mask for the model update window is constructed. The mask entropy loss is calculated based on the mask, and the low-rank adaptation parameters are updated through backpropagation.

[0011] In conjunction with the first aspect, in one implementation, extracting the word-level evidence quality based on the predicted score in the generation step includes: From the predicted scores of the lexical units, select the first preset number of lexical units with the largest values ​​as the first lexical unit set for this generation step; The evidence quality of each prediction score in the first word set is calculated using a preset positive value mapping function. Based on the aforementioned evidence quality, the word-level evidence quality of the generation step is calculated.

[0012] In conjunction with the first aspect, in one implementation, calculating the lexical-level evidence quality of the generation step based on the evidence quality includes: The quality of the evidence is converted into a preset probability distribution parameter; The word-level evidence quality is obtained by summing the preset probability distribution parameters corresponding to each predicted score of the first word set.

[0013] In conjunction with the first aspect, in one implementation, the computation of lexical-level cognitive uncertainty in the generation step includes: Based on the first preset quantity and the quality of the lexical-level evidence, the lexical-level cognitive uncertainty is obtained.

[0014] In conjunction with the first aspect, in one implementation method, smoothing out cognitive uncertainty at the word level includes: According to the order of each generation step, a preset trajectory smoothing algorithm is used to smooth the word-level cognitive uncertainty to obtain the smoothed uncertainty.

[0015] In conjunction with the first aspect, in one implementation, determining the stable region of the generation step based on smoothing uncertainty includes: Based on the aforementioned smoothing uncertainty and preset tolerance, a tolerance band is constructed for the generation step; Determine whether the smoothing uncertainty is within the tolerance band; if so, determine that the corresponding generation step is in the stable region.

[0016] In conjunction with the first aspect, in one implementation, based on the smoothing uncertainty and a preset tolerance, a tolerance band for the generation step is constructed, including: The minimum value among the smoothing uncertainties of the current generation step and the preset number of previous generation steps is determined as the running minimum value; Based on the minimum operating value and the preset tolerance, a tolerance band is constructed for the current generation step.

[0017] In conjunction with the first aspect, in one implementation, constructing a mask for the model update window includes: Starting from the first generation step, the number of generation steps with smoothed uncertainty in the stable region and consecutive generation order is counted. If the number reaches a preset patience parameter, the mask for all generation steps from the first generation step to the current generation step is set to 1, and the mask for the remaining generation steps is set to 0.

[0018] In conjunction with the first aspect, in one implementation, calculating the mask entropy loss based on the mask includes: Calculate the lexical-level conditional entropy for each generation step; The mask is used to weight the lexical-level conditional entropy to obtain the mask entropy loss.

[0019] In conjunction with the first aspect, in one implementation, the lexical-level conditional entropy of each generation step is calculated, including: The predicted scores of the words in the vocabulary at each generation step are converted into the predicted probability distribution of the corresponding generation step. The information entropy is calculated based on the predicted probability distribution and used as the lexical-level conditional entropy for each generation step.

[0020] In conjunction with the first aspect, in one implementation, after calculating the mask entropy loss based on the mask, and before updating the low-rank adaptation parameters via backpropagation, the method further includes: Calculate an entropy loss adjustment term, which includes at least one of evidence gain weighted entropy loss that weights the entropy loss based on evidence gain and evidence monotonicity regularization term that constrains the monotonicity of uncertainty changes in adjacent generation steps. Based on the mask entropy loss and the entropy loss adjustment term, a composite objective function is constructed; The step of updating the low-rank adaptation parameters through backpropagation includes: calculating the gradient based on the composite objective function and updating the low-rank adaptation parameters.

[0021] In conjunction with the first aspect, in one implementation, when the entropy loss adjustment term includes the evidence gain-weighted entropy loss, its calculation steps include: Calculate the evidence gain of adjacent generation steps based on the lexical-level cognitive uncertainty of the generation steps within the model update window; The importance score is calculated using the evidence gain, and the importance weight is obtained after normalization. Based on the importance weights and the lexical-level conditional entropy of the corresponding generation steps, the evidence gain weighted entropy loss is calculated.

[0022] In conjunction with the first aspect, in one implementation, when the entropy loss adjustment term includes the evidence monotonicity regularization term, its calculation steps include: The difference in word-level cognitive uncertainty between adjacent generation steps within the calculation model update window; A penalty term is constructed based on the difference, serving as the evidence monotonicity regularization term.

[0023] Secondly, embodiments of this application provide a large language model test-time learning device, the large language model test-time learning device comprising: The prediction score acquisition module is used to input test data into a large language model with low-rank adaptation parameters and obtain the prediction score of the word in the vocabulary at each generation step. An uncertainty acquisition module is used to extract the lexical-level evidence quality of the generation step based on the prediction score, and to calculate the lexical-level cognitive uncertainty of the generation step. The mask acquisition module is used to smooth out cognitive uncertainty at the word level, determine the stable region of the generation step based on the smoothed uncertainty, and construct the mask of the model update window. An update module is used to calculate the mask entropy loss based on the mask and update the low-rank adaptation parameters through backpropagation.

[0024] Thirdly, embodiments of this application provide a large language model test-time learning device, the large language model test-time learning device including a processor, a memory, and a large language model test-time learning program stored in the memory and executable by the processor, wherein when the large language model test-time learning program is executed by the processor, it implements the steps of the large language model test-time learning method as described in any of the preceding claims.

[0025] Fourthly, embodiments of this application provide a computer-readable storage medium storing a large language model test-time learning program, wherein when the large language model test-time learning program is executed by a processor, it implements the steps of the large language model test-time learning method as described in any of the preceding claims.

[0026] The beneficial effects of the technical solution provided in this application include: This application provides a method, apparatus, device, and medium for learning large language models during testing. The method provided in this embodiment extracts evidence quality by predicting scores and then calculates cognitive uncertainty. This avoids the ambiguity of traditional uncertainty estimation based on probability distributions (such as entropy) in autoregressive generation. It can accurately identify unreliable generated tokens, provide reliable signal support for testing-time updates, and effectively suppress adaptive drift.

[0027] The method provided in this embodiment proposes an evidence-guided dynamic token selection strategy. Based on smoothing and mask construction, it selects continuous and stable generation regions for updating, avoiding the inclusion of low-reliability tokens and redundant information in the update process, avoiding error accumulation caused by noisy steps, reducing computational overhead, enhancing the model's anti-interference ability, and improving the learning efficiency during testing.

[0028] The method provided in this embodiment uses low-rank adaptation (LoRA) technology for parameter updates, which only requires updating a lightweight subset of the model's parameters without full parameter optimization. This reduces computational and storage costs while ensuring adaptive performance, making it easier for practical deployment. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating an embodiment of the learning method for testing large language models in this application. Figure 2 This is a schematic diagram of the functional modules of an embodiment of the learning device for testing large language models according to this application; Figure 3 This is a schematic diagram of the hardware structure of the learning device for testing large language models involved in the embodiments of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] First, some of the technical terms used in this application will be explained to help those skilled in the art understand this application.

[0033] Token: The smallest unit of information that a large model processes.

[0034] Prediction scores, also known as logits, are the raw values ​​output by the last layer in a deep learning model's prediction process. They are typically an unnormalized vector of real numbers, with each value corresponding to a class. The values ​​of logits can be positive, negative, or even very large or very small.

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] Firstly, embodiments of this application provide a learning method for testing large language models.

[0037] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the learning method used in testing a large language model according to this application. Figure 1 As shown, the learning methods used during testing of large language models include: S101: Input the test data into a large language model with low-rank adaptation parameters and obtain the prediction score of the word in the vocabulary at each generation step.

[0038] In step S101, the test data needs to be preprocessed to obtain unlabeled test data. Data types include, but are not limited to, question-and-answer text, instruction text, and document fragments. The test data undergoes format standardization, including removing invalid characters, standardizing text encoding, and adding standard prompt templates (such as "Please answer the following questions: {input}"), to ensure that the input format meets the input requirements of the large language model.

[0039] Load a large language model that has been pre-trained and supervised fine-tuning.

[0040] The large language models used in this embodiment include, but are not limited to, Qwen2.5-7B-Instruct, Llama3.1-8B-Instruct, and Phi-4-14B.

[0041] Understandably, the specific parameter values ​​can be set according to actual needs when initializing parameters for a large language model. For example, as an example, initialize the low-rank adaptation (LoRA) parameter ΔΘ=BA, where A is initialized using a random Gaussian distribution (mean 0, variance 0.01), and B is initialized as a matrix of all zeros; set the rank of LoRA to 8, and the adapted network layer to be the attention layer of the Transformer encoder. Set the hyperparameters learned during testing: the number of the first K Logits extracted from evidence K=5, the smoothing coefficient γ=0.1, and the absolute tolerance δ. abs =0.01, relative tolerance δ rel =0.05, patience parameter P=3, balance coefficient α=0.5, weight parameter λ EGWE =0.3, λ EMR =0.1, use the AdamW optimizer to update ΔΘ, with a learning rate of 5e 5 The batch size is 1.

[0042] The standardized test data is input into the large language model, which generates a token sequence using an autoregressive approach. At each generation step, the large language model calculates the logits z of the next-token based on the input test data and the previously generated token sequence. t ∈ V(V is the size of the model vocabulary, representing the total number of lexical units in the vocabulary. For example, the vocabulary size of Qwen2.5-7B-Instruct is 151936.) Thus, in each generation step, the large language model calculates the prediction score of each lexical unit in its vocabulary, and these prediction scores are used as the output logits.

[0043] For example, taking the vocabulary of Qwen2.5-7B-Instruct as an example, large language models will calculate the prediction scores of these 151,936 tokens, obtain 151,936 values, and form a vector logits.

[0044] It is understandable that the word units corresponding to the predicted scores mentioned above do not include the word units corresponding to the prompt words, such as those corresponding to the test data.

[0045] S102: Extract the lexical-level evidence quality of the generation step based on the predicted score, and calculate the lexical-level cognitive uncertainty of the generation step.

[0046] S103: Smooth the cognitive uncertainty at the word level, determine the stable region of the generation step based on the smoothed uncertainty, and construct the mask of the model update window.

[0047] S104: Calculate the mask entropy loss based on the mask, and update the low-rank adaptation parameters through backpropagation.

[0048] The method provided in this embodiment extracts evidence quality by predicting scores and then calculates cognitive uncertainty. This avoids the ambiguity of traditional uncertainty estimation based on probability distributions (such as entropy) in autoregressive generation. It can accurately identify unreliable generated tokens, provide reliable signal support for test-time updates, and effectively suppress adaptive drift.

[0049] The method provided in this embodiment proposes an evidence-guided dynamic token selection strategy. Based on smoothing and mask construction, it selects continuous and stable generation regions for updating, avoiding the inclusion of low-reliability tokens and redundant information in the update process, avoiding error accumulation caused by noisy steps, reducing computational overhead, enhancing the model's anti-interference ability, and improving the learning efficiency during testing.

[0050] The method provided in this embodiment uses low-rank adaptation (LoRA) technology for parameter updates, which only requires updating a lightweight subset of the model's parameters without full parameter optimization. This reduces computational and storage costs while ensuring adaptive performance, making it easier for practical deployment.

[0051] Furthermore, in one embodiment, the word-level evidence quality extracted based on the predicted score generation step includes: S201: Select the first preset number of words with the largest predicted scores from the word scores as the first word set for this generation step.

[0052] In step S201, the first preset quantity K can be set according to the actual situation. Generally, it can be a positive integer from 1 to 50, with a preferred range of 3 to 10. For example, taking the vocabulary of Qwen2.5-7B-Instruct as an example, if the first preset quantity K=5, for generation step t, the prediction scores of its 151936 tokens can be sorted from largest to smallest, and then the top 5 tokens are selected as the first word set for generation step t. These 5 tokens are the tokens with the largest prediction scores relative to the other tokens.

[0053] Understandably, the token with the highest prediction score in the first token set will be used as the answer for generation step t and output.

[0054] S202: Calculate the evidence quality of each prediction score of the first word set using a preset positive value mapping function.

[0055] In step S202, the positive value mapping function can be softplus, ReLU, Exp, Sigmoid, etc. The appropriate function can be selected based on actual needs.

[0056] For example, the softplus function is used in this embodiment.

[0057] For the first word set obtained in step S201, its prediction score is denoted as z. t (K), using the softplus function to calculate the quality of evidence e t (K) = softplus(z) t (K)).

[0058] As an example, the first preset quantity K=5, z t (5) = [2.3, 1.8, 1.5, 1.2, 0.9], then e t (5) = softplus(z) t (5))=[softplus(2.3), softplus(1.8), softplus(1.5), softplus(1.2), softplus(0.9)]=[4.03, 2.89, 2.31, 1.86, 1.53].

[0059] S203: Based on the evidence quality, calculate the word-level evidence quality of the generation step.

[0060] This embodiment selects the highest-valued first-preset-quantity word from the predicted scores, focusing on the high-confidence candidate set. This effectively filters out noise interference from low-scoring words in the vocabulary, reducing computational complexity while improving the targeting of evidence extraction. By using a preset positive value mapping function to transform the predicted scores into evidence quality, the non-negativity of the evidence parameters is ensured. This solves the problem of evidence extraction being greatly affected by noise and having an uncontrollable numerical range, resulting in more reliable quality data. This provides a solid basis for credibility in subsequent model updates, improving the accuracy and effectiveness of evidence extraction.

[0061] Further, in one embodiment, calculating the lexical-level evidence quality of the generation step based on the evidence quality includes: S301: Convert the quality of the evidence into a preset probability distribution parameter.

[0062] In step S301, the Dirichlet distribution can be used.

[0063] Specifically, the quality of evidence e t (K) is converted to Dirichlet distribution parameters: a t (K)=e t (K)+1.

[0064] For example, a t (5)=e t (5)+1=[5.03, 3.89, 3.31, 2.86, 2.53].

[0065] S302: Summing the preset probability distribution parameters corresponding to each prediction score of the first lexical set yields the lexical-level evidence quality.

[0066] In step S302, the above a t (5) Taking this as an example, calculate the word-level evidence quality Z. zt (5): Z zt (5)= .

[0067] This embodiment achieves standardized and probabilistic representation of evidence by converting evidence quality into probability distribution parameters. By summing the parameters of the first lexical set, evidence information from multiple high-confidence lexical units is aggregated, avoiding the one-sidedness of evidence from a single lexical unit. This solves the technical problem of difficulty in quantifying the overall evidence strength of the generation step, obtaining lexical-level evidence quality that can characterize the overall credibility of the step. This provides an accurate cumulative evidence basis for subsequent processing, improving the comprehensiveness and reliability of evidence evaluation.

[0068] Furthermore, in one embodiment, calculating the lexical-level cognitive uncertainty in the generation step includes: obtaining the lexical-level cognitive uncertainty based on the first preset quantity and the quality of the lexical-level evidence.

[0069] Specifically, with the first preset quantity as K, the word-level cognitive uncertainty U t (K) is calculated as follows: U t (K) = K / Z zt (K) For example, taking the above a t (5) For example, U t (5) = 5 / Z zt (5) = 5 / ≈0.284.

[0070] Understandably, for cue words such as test data and [EOS] token, the word-level cognitive uncertainty is 0.

[0071] This embodiment calculates cognitive uncertainty based on a preset quantity and evidence quality, quantifies evidence strength as uncertainty, solves the problem that traditional methods are difficult to accurately measure uncertainty based on evidence quality, and obtains a reliable index reflecting the cognitive confidence of the model, providing a basis for subsequent screening of stable regions and improving the accuracy of the evaluation.

[0072] Furthermore, in one embodiment, smoothing the lexical-level cognitive uncertainty includes: smoothing the lexical-level cognitive uncertainty using a preset trajectory smoothing algorithm according to the order of each generation step, so as to eliminate random fluctuations in the lexical-level cognitive uncertainty between adjacent generation steps, so as to obtain the smoothed uncertainty.

[0073] This embodiment uses the exponential moving average algorithm. In other embodiments, it can also be any smoothing algorithm that can eliminate time series fluctuations, such as the moving average algorithm or the low-pass filtering algorithm.

[0074] For example, assuming there are M generation steps, i.e., t=1,2,...,M, then a total of M tokens need to be generated and output, where M is a positive integer.

[0075] For generation step t=1, the smoothed uncertainty can generally be considered equal to the word-level cognitive uncertainty, i.e. =U1(K).

[0076] For the generation steps t=2, ...,M, smooth the uncertainty. The calculation is as follows: =(1-γ)× +γ×U t(K) γ is a smoothing coefficient, 0 < γ ≤ 1, with a preferred range of 0.05 to 0.3.

[0077] This embodiment specifically employs the exponential moving average algorithm to smooth the trajectory of the uncertain sequence according to the generation order. By fusing historical step information, it effectively eliminates noise from single-step random fluctuations. This solves the problem of misjudging stable regions caused by severe jitter in the original uncertainty curve. A smooth and continuous uncertainty trajectory is obtained, avoiding erroneous updates caused by accidental fluctuations, improving the accuracy and robustness of subsequent stable region judgments, ensuring that the model learns only on truly reliable continuous segments, and significantly enhancing the stability of the model's adaptation process.

[0078] Furthermore, in one embodiment, determining the stable region of the generation step based on smoothing uncertainty includes: S401: Based on the smoothing uncertainty and the preset tolerance, construct the tolerance band of the generation step.

[0079] S402: Determine whether the smoothing uncertainty is within the tolerance band. If so, determine that the corresponding generation step is in the stable region.

[0080] In step S401, the tolerance band for step t is generated as band(t). In step S402, during the judgment, if the smoothing uncertainty of step t is... If band(t) is less than or equal to band(t), then the generation step t is in a stable region; otherwise, the generation step t is not in a stable region.

[0081] This embodiment constructs a tolerance band by combining smoothed uncertainty with a preset tolerance, achieving dynamic adaptation of the stability judgment criterion and enhancing judgment fault tolerance. By determining whether the smoothed uncertainty lies within the tolerance band, stable regions are accurately screened, solving the problem of misjudgment of stable regions due to improper threshold setting. This yields highly reliable stable region division results, ensuring that the model updates only for credible steps, effectively shielding noise interference, and significantly improving the accuracy and robustness of learning during testing.

[0082] Furthermore, in one embodiment, based on the smoothing uncertainty and a preset tolerance, a tolerance band for the generation step is constructed, including: S501: Determine the minimum value among the smoothing uncertainties of the current generation step and the preset number of previous generation steps as the running minimum value.

[0083] Still assuming there are M generation steps, t=1,2,...,M, in step S501, the smoothing uncertainty of the M generation steps can be obtained: , , , , ... .

[0084] For the current generation step t, the minimum value is denoted as... : =min( ), t0=1,2,...,t.

[0085] For example, if t=5, then =min( ).

[0086] The number of preceding steps can be set according to actual needs. For example, in this embodiment, the number of preceding steps is the total number of the current generation step and all generation steps before the current generation step.

[0087] S502: Based on the minimum operating value and the preset tolerance, construct the tolerance band for the current generation step.

[0088] In step S502, the tolerance band (t) of step t is generated as follows: band(t) = δ abs +δ rel × +

[0089] Preset tolerance includes absolute tolerance δ abs and relative tolerance δ rel At least one of them, δ abs The value is generally taken as 0 to 0.1, with an preferred range of 0.001 to 0.05, δ rel The value is generally between 0 and 1, with an preferred range of 0.01 to 0.2.

[0090] This embodiment selects the minimum value of smoothed uncertainty from the current and previous historical steps as the running minimum, and uses the lowest uncertainty within a local window as a dynamic benchmark. Based on this benchmark and tolerance, a tolerance band is constructed, enabling adaptive adjustment of the judgment criteria. This solves the technical problem that fixed thresholds are difficult to adapt to local data fluctuations and single-point outlier interference. A dynamically changing elastic tolerance range is obtained as the generation process progresses, effectively avoiding misjudgments of stable regions due to accidental fluctuations, ensuring the accuracy and robustness of the model update strategy, and improving the stability of the adaptation process.

[0091] Furthermore, in one embodiment, constructing the mask for the model update window includes: S601: Starting from the first generation step, scan and count the number of generation steps where the smoothing uncertainty is in the stable region and the generation order is consecutive; S602: If the quantity reaches a preset patience parameter, the mask of all generation steps from the first generation step to the current generation step is set to 1, and the mask of the remaining generation steps is set to 0.

[0092] Steps S601-S602 involve dynamically updating the window construction, initializing the stability counter c=0, and starting the scan from t=1: if the current generation step is in a stable region, c is incremented; otherwise, c is reset to 0. For example, when scanning to t=t... c (t) c When ≤M), c reaches the patience parameter P=3, that is, t=t c -2、t=t c -1、t=t c If three consecutive generation steps are in a stable region, then set the update window length C. K =t c The token index for the corresponding generation step in the updated window is {t=1,...,t=t}. c}. Construct mask s t : t∈{1,2,...,t c} time s t =1, the rest at time t s t =0.

[0093] For example, as example t c =12, then update the window length C K =12, update the token index of the corresponding generation step in the window to {t=1,2,...,t=12}. Construct mask s t When t∈{1,2,...,12}, s t =1, the rest at time t s t =0.

[0094] The patience parameter can be preset according to actual needs, and is generally a positive integer from 1 to 10, with the preferred range being 2 to 5.

[0095] This embodiment identifies consistently reliable segments by scanning and counting the number of consecutive stable steps. Using a patience parameter threshold, the mask is only updated when the number of consecutive stable steps reaches a certain threshold. This solves the problem of erroneous updates caused by brief periods of stability or noise fluctuations. It yields a learning mask specifically for long, continuous stable regions, avoiding error accumulation caused by fragmented updates. This ensures the model adjusts parameters only in highly reliable contexts, significantly improving the stability and robustness of learning during testing and preventing model performance degradation.

[0096] Furthermore, in one embodiment, calculating the mask entropy loss based on the mask includes: S701: Calculate the lexical-level conditional entropy for each generation step; S702: The term-level conditional entropy is weighted using the mask to obtain the mask entropy loss.

[0097] In step S702, taking generation step t as an example, its word-level conditional entropy is denoted as H. t .

[0098] Mask entropy loss .

[0099] For example, t c =12, .

[0100] This embodiment quantifies the prediction uncertainty at each step by calculating word-level conditional entropy and uses a mask to weight the entropy, thereby achieving region selection for loss calculation. This solves the problem of model performance degradation caused by noisy steps interfering with gradient updates. It yields an objective function based solely on stable regions, avoiding unreliable data from misleading parameter updates, ensuring model optimization on high-confidence segments, significantly improving the stability, accuracy, and robustness of learning during testing, and preventing model degradation.

[0101] Further, in one embodiment, the lexical-level conditional entropy of each generation step is calculated, including: S801: Convert the prediction scores of the words in the vocabulary at each generation step into the prediction probability distribution of the corresponding generation step; In this embodiment, the softmax function is used for transformation to calculate the predicted probability distribution of lexical units in the vocabulary at each generation step. Other normalization methods (such as sigmoid or custom normalization functions) can also be used.

[0102] As an example, =softmax( ), where v=1,2,...,V, and V is the total number of lexical units in the model vocabulary. To generate the prediction score for the v-th word in step t, The predicted probability distribution of the v-th word in step t.

[0103] S802: Calculate the information entropy based on the predicted probability distribution, as the word-level conditional entropy for each generation step.

[0104] In this embodiment, the Shannon entropy formula is used to calculate information entropy. However, in other embodiments, other normalization functions or equivalent information entropy calculation methods may also be used.

[0105] Specifically, the lexical-level conditional entropy of step t is generated. .

[0106] This embodiment achieves standardization by transforming predicted scores into a probability distribution. Information entropy is calculated based on the distribution to quantify the uncertainty of single-step prediction. This solves the problem of difficulty in measuring prediction disorder, obtaining a word-level conditional entropy index, providing a reliable basis for loss calculation, and effectively improving the model's adaptive accuracy.

[0107] Furthermore, in one embodiment, after calculating the mask entropy loss based on the mask, and before updating the low-rank adaptation parameters via backpropagation, the method further includes: S901: Calculate the entropy loss adjustment term, which includes at least one of the following: evidence gain weighted entropy loss that weights the entropy loss based on the evidence gain and evidence monotonicity regularization term that constrains the monotonicity of uncertainty changes in adjacent generation steps. S902: Construct a composite objective function based on the mask entropy loss and the entropy loss adjustment term; In step S902, the composite objective function .

[0108] in, For mask entropy loss, For evidence gain weighted entropy loss, λ EGWE The weighting parameters are the evidence gain weighted entropy parameters. λ is the evidence monotonicity regularity term. EMR λ is the weight parameter for the evidence monotonicity regularization term. EGWE and λ EMR It can be set as needed, and the value range is 0 to 1.

[0109] The step of updating the low-rank adaptation parameters through backpropagation includes: calculating the gradient based on the composite objective function and updating the low-rank adaptation parameters.

[0110] This embodiment constructs a composite objective function by introducing an entropy loss adjustment term. It utilizes evidence gain-weighted entropy loss, weighted according to evidence reliability, to address the problem of low-quality data interference caused by traditional loss methods that treat all steps equally, allowing the model to focus on optimizing high-evidence-quality regions. By using an evidence monotonicity regularization term to constrain uncertainty changes, it addresses the problem of drastic fluctuations in uncertainty between adjacent steps, ensuring a smooth and continuous adaptation process. This results in a composite function with multi-objective collaborative optimization, significantly improving the accuracy, stability, and robustness of model updates and effectively preventing model performance degradation.

[0111] Further, in one embodiment, when the entropy loss adjustment term includes the evidence gain-weighted entropy loss, its calculation steps include: S1001: Calculate the evidence gain of adjacent generation steps based on the lexical-level cognitive uncertainty of the generation steps within the model update window.

[0112] Among them, evidence gain m=2, ...,t c .

[0113] And when m=1, .

[0114] for example,

[0115] = - =0.008.

[0116] S1002: Calculate the importance score using the evidence gain, and obtain the importance weight after normalization.

[0117] Importance score

[0118] α is a balance coefficient, which can be set manually. It is generally set to 0 to 1, with an optimal range of 0.3 to 0.7.

[0119] After normalizing the importance scores, the importance weights are obtained. Normalizing importance scores means making the mean of the importance scores equal to 1. ,for example =12, making That is, the mean is 1.

[0120] S1003: Based on the importance weights and the lexical-level conditional entropy of the corresponding generation steps, the evidence gain weighted entropy loss is calculated.

[0121] Evidence gain weighted entropy loss = .

[0122] Furthermore, in one embodiment, when the entropy loss adjustment term includes the evidence monotonicity regularization term, its calculation steps include: S1101: Calculate the difference in word-level cognitive uncertainty between adjacent generation steps within the model update window.

[0123] S1102: Construct a penalty term based on the difference, which serves as the evidence monotonicity regularization term.

[0124] calculate

[0125] If adjacent pairs Then the item The value is 0, which only penalizes situations where uncertainty increases.

[0126] This embodiment enhances the contribution of effective information to model updates by using evidence gain weighting and evidence monotonicity regularization, while suppressing the fluctuation transmission of uncertainty in the autoregressive generation process, thereby improving the adaptive stability and generalization performance of the model in distributed offset scenarios.

[0127] Secondly, embodiments of this application also provide a learning device for testing large language models.

[0128] In one embodiment, reference is made to Figure 2 , Figure 2 This is a schematic diagram of the functional modules of an embodiment of the learning device for testing large language models according to this application. Figure 2 As shown, the learning device for testing the large language model includes: The prediction score acquisition module is used to input test data into a large language model with low-rank adaptation parameters and obtain the prediction score of the word in the vocabulary at each generation step. An uncertainty acquisition module is used to extract the lexical-level evidence quality of the generation step based on the prediction score, and to calculate the lexical-level cognitive uncertainty of the generation step. The mask acquisition module is used to smooth out cognitive uncertainty at the word level, determine the stable region of the generation step based on the smoothed uncertainty, and construct the mask of the model update window. An update module is used to calculate the mask entropy loss based on the mask and update the low-rank adaptation parameters through backpropagation.

[0129] Furthermore, in one embodiment, the uncertainty acquisition module extracts the word-level evidence quality of the generation step based on the prediction score, including: From the predicted scores of the lexical units, select the first preset number of lexical units with the largest values ​​as the first lexical unit set for this generation step; The evidence quality of each prediction score in the first word set is calculated using a preset positive value mapping function. Based on the aforementioned evidence quality, the word-level evidence quality of the generation step is calculated.

[0130] Furthermore, in one embodiment, the uncertainty acquisition module calculates the word-level evidence quality of the generation step based on the evidence quality, including: The quality of the evidence is converted into a preset probability distribution parameter; The word-level evidence quality is obtained by summing the preset probability distribution parameters corresponding to each predicted score of the first word set.

[0131] Furthermore, in one embodiment, the uncertainty acquisition module calculates the lexical-level cognitive uncertainty of the generation step, including: Based on the first preset quantity and the quality of the lexical-level evidence, the lexical-level cognitive uncertainty is obtained.

[0132] Furthermore, in one embodiment, the mask acquisition module performs smoothing processing on word-level cognitive uncertainty, including: According to the order of each generation step, a preset trajectory smoothing algorithm is used to smooth the word-level cognitive uncertainty to obtain the smoothed uncertainty.

[0133] Furthermore, in one embodiment, the mask acquisition module determines the stable region of the generation step based on smoothing uncertainty, including: Based on the aforementioned smoothing uncertainty and preset tolerance, a tolerance band is constructed for the generation step; Determine whether the smoothing uncertainty is within the tolerance band; if so, determine that the corresponding generation step is in the stable region.

[0134] Furthermore, in one embodiment, the mask acquisition module constructs a tolerance band for the generation step based on the smoothing uncertainty and a preset tolerance, including: The minimum value among the smoothing uncertainties of the current generation step and the preset number of previous generation steps is determined as the running minimum value; Based on the minimum operating value and the preset tolerance, a tolerance band is constructed for the current generation step.

[0135] Furthermore, in one embodiment, the mask acquisition module constructs the mask for the model update window, including: Starting from the first generation step, the number of generation steps with smoothed uncertainty in the stable region and consecutive generation order is counted. If the quantity reaches a preset patience parameter, the mask of the generation step corresponding to the quantity is set to 1, and the mask of the other generation steps is set to 0.

[0136] Further, in one embodiment, the update module calculates the mask entropy loss based on the mask, including: Calculate the lexical-level conditional entropy for each generation step; The mask is used to weight the lexical-level conditional entropy to obtain the mask entropy loss.

[0137] Furthermore, in one embodiment, the update module calculates the lexical-level conditional entropy for each generation step, including: The predicted scores of the words in the vocabulary at each generation step are converted into the predicted probability distribution of the corresponding generation step. The information entropy is calculated based on the predicted probability distribution and used as the lexical-level conditional entropy for each generation step.

[0138] Furthermore, in one embodiment, after calculating the mask entropy loss based on the mask, and before updating the low-rank adaptation parameters via backpropagation, the update module is further configured to: Calculate an entropy loss adjustment term, which includes at least one of evidence gain weighted entropy loss that weights the entropy loss based on evidence gain and evidence monotonicity regularization term that constrains the monotonicity of uncertainty changes in adjacent generation steps. Based on the mask entropy loss and the entropy loss adjustment term, a composite objective function is constructed; The update module updates the low-rank adaptation parameters through backpropagation, including: calculating the gradient based on the composite objective function and updating the low-rank adaptation parameters.

[0139] Further, in one embodiment, when the entropy loss adjustment term includes the evidence gain-weighted entropy loss, the update module calculates the entropy loss adjustment term as follows: Calculate the evidence gain of adjacent generation steps based on the lexical-level cognitive uncertainty of the generation steps within the model update window; The importance score is calculated using the evidence gain, and the importance weight is obtained after normalization. Based on the importance weights and the lexical-level conditional entropy of the corresponding generation steps, the evidence gain weighted entropy loss is calculated.

[0140] Further, in one embodiment, when the entropy loss adjustment term includes the evidence monotonicity regularization term, the update module calculates the entropy loss adjustment term as follows: The difference in word-level cognitive uncertainty between adjacent generation steps within the calculation model update window; A penalty term is constructed based on the difference, serving as the evidence monotonicity regularization term.

[0141] The functions of each module in the above-mentioned large language model testing learning device correspond to the steps in the above-mentioned large language model testing learning method embodiment, and their functions and implementation processes will not be described in detail here.

[0142] Thirdly, embodiments of this application provide a large language model learning device for testing. The large language model learning device for testing can be a personal computer (PC), a laptop, a server, or other device with data processing capabilities.

[0143] Reference Figure 3 , Figure 3 This is a schematic diagram of the hardware structure of the large language model test-time learning device involved in the embodiments of this application. In the embodiments of this application, the large language model test-time learning device may include a processor, memory, communication interface, and communication bus.

[0144] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0145] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of the learning device during large language model testing, as well as interfaces used for interconnecting the learning device with other devices (such as other computing devices or user equipment) during large language model testing. Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0146] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0147] The processor can be a general-purpose processor, which can call the large language model test-time learning program stored in memory and execute the large language model test-time learning method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the large language model test-time learning program is called can be referred to in the various embodiments of the large language model test-time learning method of this application, and will not be repeated here.

[0148] Those skilled in the art will understand that Figure 3 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0149] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0150] The present application stores a large language model test-time learning program on a computer-readable storage medium, wherein when the large language model test-time learning program is executed by a processor, it implements the steps of the large language model test-time learning method as described above.

[0151] The method implemented when the large language model test-time learning program is executed can be referred to in the various embodiments of the large language model test-time learning method of this application, and will not be repeated here.

[0152] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0153] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0154] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0155] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0156] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0158] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A learning method for large language models during testing, characterized in that, The learning methods for testing the large language model include: Input the test data into a large language model with low-rank adaptation parameters and obtain the prediction score of the word in the vocabulary at each generation step; Based on the predicted score, extract the lexical-level evidence quality of the generation step and calculate the lexical-level cognitive uncertainty of the generation step. The uncertainty at the word level is smoothed, the stable region of the generation step is determined based on the smoothed uncertainty, and a mask for the model update window is constructed. The construction of the mask for the model update window includes: scanning from the first generation step and counting the number of generation steps whose smoothed uncertainty is in the stable region and whose generation order is consecutive; if the number reaches a preset patience parameter, the mask of all generation steps from the first generation step to the current generation step is set to 1, and the mask of the remaining generation steps is set to 0. Based on the mask, calculate the mask entropy loss, and calculate the entropy loss adjustment term, which includes at least one of the evidence gain weighted entropy loss that weights the entropy loss based on the evidence gain and the evidence monotonicity regularization term that constrains the monotonicity of uncertainty changes in adjacent generation steps; based on the mask entropy loss and the entropy loss adjustment term, construct a composite objective function; calculate the gradient based on the composite objective function, and update the low-rank adaptation parameters.

2. The large language model test-time learning method as described in claim 1, characterized in that, The word-level evidence quality based on the predicted score extraction generation step includes: From the predicted scores of the lexical units, select the first preset number of lexical units with the largest values ​​as the first lexical unit set for this generation step; The evidence quality of each prediction score in the first word set is calculated using a preset positive value mapping function. Based on the aforementioned evidence quality, the word-level evidence quality of the generation step is calculated.

3. The large language model test-time learning method as described in claim 2, characterized in that, Based on the aforementioned evidence quality, the word-level evidence quality of the generation step is calculated, including: The quality of the evidence is converted into a preset probability distribution parameter; The word-level evidence quality is obtained by summing the preset probability distribution parameters corresponding to each predicted score of the first word set.

4. The large language model test-time learning method as described in claim 2, characterized in that, The computational generation step involves lexical-level cognitive uncertainty, including: Based on the first preset quantity and the quality of the lexical-level evidence, the lexical-level cognitive uncertainty is obtained.

5. The large language model test-time learning method as described in claim 1, characterized in that, Smoothing of cognitive uncertainty at the word level includes: According to the order of each generation step, a preset trajectory smoothing algorithm is used to smooth the word-level cognitive uncertainty to obtain the smoothed uncertainty.

6. The large language model test-time learning method as described in claim 1, characterized in that, The stable region of the generation step is determined based on the smoothing uncertainty, including: Based on the aforementioned smoothing uncertainty and preset tolerance, a tolerance band is constructed for the generation step; Determine whether the smoothing uncertainty is within the tolerance band; if so, determine that the corresponding generation step is in the stable region.

7. The large language model test-time learning method as described in claim 6, characterized in that, Based on the aforementioned smoothing uncertainty and a preset tolerance, a tolerance band is constructed for the generation step, including: The minimum value among the smoothing uncertainties of the current generation step and the preset number of previous generation steps is determined as the running minimum value; Based on the minimum operating value and the preset tolerance, a tolerance band is constructed for the current generation step.

8. The large language model test-time learning method as described in claim 1, characterized in that, Calculating the mask entropy loss based on the mask includes: Calculate the lexical-level conditional entropy for each generation step; The mask is used to weight the lexical-level conditional entropy to obtain the mask entropy loss.

9. The large language model test-time learning method as described in claim 8, characterized in that, Calculate the lexical-level conditional entropy for each generation step, including: The predicted scores of the words in the vocabulary at each generation step are converted into the predicted probability distribution of the corresponding generation step. The information entropy is calculated based on the predicted probability distribution and used as the lexical-level conditional entropy for each generation step.

10. The large language model test-time learning method as described in claim 1, characterized in that, When the entropy loss adjustment term includes the evidence gain-weighted entropy loss, its calculation steps include: Calculate the evidence gain of adjacent generation steps based on the lexical-level cognitive uncertainty of the generation steps within the model update window; The importance score is calculated using the evidence gain, and the importance weight is obtained after normalization. Based on the importance weights and the lexical-level conditional entropy of the corresponding generation steps, the evidence gain weighted entropy loss is calculated.

11. The large language model test-time learning method as described in claim 1, characterized in that, When the entropy loss adjustment term includes the evidence monotonicity regularization term, its calculation steps include: The difference in word-level cognitive uncertainty between adjacent generation steps within the calculation model update window; A penalty term is constructed based on the difference, serving as the evidence monotonicity regularization term.

12. A learning device for testing a large language model, characterized in that, The large language model learning device for testing includes: The prediction score acquisition module is used to input test data into a large language model with low-rank adaptation parameters and obtain the prediction score of the word in the vocabulary at each generation step. An uncertainty acquisition module is used to extract the lexical-level evidence quality of the generation step based on the prediction score, and to calculate the lexical-level cognitive uncertainty of the generation step. The mask acquisition module is used to smooth the cognitive uncertainty at the word level, determine the stable region of the generation steps based on the smoothed uncertainty, and construct a mask for the model update window. Constructing the mask for the model update window includes: scanning from the first generation step and counting the number of generation steps whose smoothed uncertainty is in the stable region and whose generation order is consecutive; if the number reaches a preset patience parameter, the mask of all generation steps from the first generation step to the current generation step is set to 1, and the mask of the remaining generation steps is set to 0. An update module is used to calculate the mask entropy loss based on the mask, calculate the entropy loss adjustment term, the entropy loss adjustment term includes at least one of evidence gain weighted entropy loss that weights the entropy loss based on evidence gain and evidence monotonicity regularization term that constrains the monotonicity of uncertainty changes in adjacent generation steps; construct a composite objective function based on the mask entropy loss and the entropy loss adjustment term; calculate the gradient based on the composite objective function, and update the low-rank adaptation parameters.

13. A learning device for testing a large language model, characterized in that, The large language model test-time learning device includes a processor, a memory, and a large language model test-time learning program stored in the memory and executable by the processor, wherein when the large language model test-time learning program is executed by the processor, it implements the steps of the large language model test-time learning method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a large language model test-time learning program, wherein when the large language model test-time learning program is executed by a processor, it implements the steps of the large language model test-time learning method as described in any one of claims 1 to 11.

Images