Method and device for calibrating language model

Abstract

The present disclosure provides a computer-implemented method for calibrating a language model (LM), the method comprising: receiving, by a first LM, a set of language questions and generating a first set of predicted answer confidence distributions respectively corresponding to the set of language questions; receiving, by the second LM, the set of language questions and generating a second set of predicted answer confidence distributions respectively corresponding to the set of language questions, where generating the second set of predicted answer confidence distributions includes calibrating the second set of predicted answer confidence distributions based on temperature scaling coefficients; determining a distance between the first set of predicted answer confidence distributions and the second set of predicted answer confidence distributions; and updating the temperature scaling factor based on the distance.

Description

Technical Field

[0001] The various aspects of this disclosure generally relate to artificial intelligence (AI), and more specifically, to methods and apparatus for calibrating language models (LM). Background Technology

[0002] Uncertainty calibration, as an important metric for building reliable deep learning systems, measures the consistency between the posterior probability (or prediction confidence) of the output given by the model and the actual probability of it being correct. For example, when a well-calibrated model gives some predictions with 80% confidence, these predictions should be accurate to 80%, meaning the model knows what it knows.

[0003] Calibrating prediction confidence helps in the application of LM. This property allows human users to better detect and correct undesirable behaviors, such as hallucinations in LM, by receiving signals from the model about the correctness of the answer and the reliability of the answer itself, thereby building trust in LM-based applications.

[0004] Aligning pre-trained learning models (LMs) with human feedback has achieved great success in real-world applications. However, compared to pre-trained LMs, known-aligned LMs are overconfident in their answers, increasing the difficulty of distinguishing between the model's true and illusory answers, thus hindering the application of aligned LMs in safety-critical fields. Summary of the Invention

[0005] To address the aforementioned issues, this disclosure proposes a simple, effective, and sampling-efficient temperature scaling method that effectively calibrates the aligned LMS in practical applications using the prediction distribution of a pre-trained LM.

[0006] According to one embodiment, this disclosure provides a computer-implemented method for calibrating a programming logic (LM), the method comprising: receiving a set of language questions by a first LM and generating a first set of predicted answer confidence distributions corresponding to the set of language questions; receiving the set of language questions by a second LM and generating a second set of predicted answer confidence distributions corresponding to the set of language questions, wherein generating the second set of predicted answer confidence distributions includes calibrating the second set of predicted answer confidence distributions based on a temperature scaling factor; determining a distance between the first set of predicted answer confidence distributions and the second set of predicted answer confidence distributions; and updating the temperature scaling factor based on the distance.

[0007] According to one embodiment, this disclosure provides a computer-implemented method for performing a language task, the method comprising: receiving a language question by an LM obtained by using a calibration method according to aspects of this disclosure; and outputting an answer in response to the language question by the LM.

[0008] According to one embodiment, this disclosure provides a computer system including one or more processors and one or more storage devices, the storage devices storing computer-executable instructions that, when executed, cause the one or more processors to perform operations of the methods described above and operations of methods according to various aspects of this disclosure.

[0009] According to one embodiment, this disclosure provides one or more computer-readable storage media storing computer-executable instructions that, when executed, cause one or more processors to perform the operations of the methods described above and to perform the operations of methods according to various aspects of this disclosure.

[0010] According to one embodiment, this disclosure provides a computer program product including computer-executable instructions that, when executed, cause one or more processors to perform the operations of the methods described above and to perform the operations of methods according to various aspects of this disclosure. Attached Figure Description

[0011] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are for illustrative purposes and not for limiting the scope of the disclosure.

[0012] Figure 1 This is a schematic block diagram illustrating pre-trained LMs and aligned LMs according to various aspects of this disclosure.

[0013] Figure 2 Exemplary language problems according to aspects of this disclosure are shown.

[0014] Figure 3 Exemplary reliability diagrams are shown according to various aspects of this disclosure.

[0015] Figure 4 Exemplary experimental results for accuracy and ECE in accordance with various aspects of this disclosure are shown.

[0016] Figure 5 Exemplary variations of experimental results for accuracy, ECE, and confidence levels under different settings are shown according to various aspects of this disclosure.

[0017] Figure 6 An exemplary process for calibrating an aligned LM is shown according to various aspects of this disclosure.

[0018] Figure 7 Exemplary calibration results are shown using different methods according to various aspects of this disclosure.

[0019] Figure 8 An exemplary process for calibrating an LM is shown according to various aspects of this disclosure.

[0020] Figure 9 Exemplary processes for performing language tasks according to various aspects of this disclosure are shown.

[0021] Figure 10 An exemplary computing system according to various aspects of this disclosure is shown. Detailed Implementation

[0022] This disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and thus implement embodiments of this disclosure, and not to impose any limitation on the scope of this disclosure.

[0023] Various embodiments will be described in detail with reference to the accompanying drawings. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. References to specific examples and embodiments are for illustrative purposes and are not intended to limit the scope of this disclosure.

[0024] Figure 1 This is a schematic block diagram illustrating pre-trained LMs and aligned LMs according to various aspects of this disclosure.

[0025] The pre-trained LM110 can be represented as Given a length of L Text token sequence x Pre-trained LM Given all previous tokens In the case of lexical space V Upper position l Token at the location conditional probability Modeling will be performed. In practical applications or scenarios, the LM 110 is expected to be based on some downstream tasks. Human instructions on (such as question-and-answer or code completion) x Generate high-quality responses y For a pre-trained LM 110, this task can be accomplished through in-context learning (ICL), where the LM learns how to perform the task by following few-shot demonstrations within the same task. Specifically, the instruction-response pairs of the downstream task are represented as... ICL passed Give the response y to instruction x, where It is a set of K independent instruction response pairs sampled from downstream tasks. The cascading of K independent instruction response pairs is used as a demonstration. For example, as... Figure 2 As shown, in the context of Example 220, LM 110 is allowed to complete ICL.

[0026] While ICL enables pre-trained LMs to be effectively adapted to downstream tasks, recent research suggests that further aligning LMs with high-quality human preference data can significantly improve their performance and generalization ability in downstream applications. Figure 1 As shown, an alignment process 130 can be performed on the pre-trained LM 110 to obtain an aligned LM 120. The alignment process 130 may include supervised fine-tuning (SFT) and / or learning from pairwise feedback (LPF). The alignment process 130 can be implemented using existing methods. In one embodiment, the alignment process 130 includes an SFT stage and an LPF stage. For the SFT stage, the language modeling objective is to maximize... The pre-trained LM in the corresponding instructions x The length is Planning response y Fine-tuning is performed on top of this. The LM obtained from the pre-trained LM through SFT can be represented as... For the LPF stage, a reward function based on implicit human preferences is used. Collect pairwise preference data ,in Then, the SFT model was further optimized. Alignment with paired data can be achieved through reinforcement learning (RLHF) strategies based on human feedback or preference-based cross-entropy loss. The resulting LPF model can be represented as... This is an example of an aligned LM 120. For example, Llama is a known pre-trained LM, and Vicuna is an aligned LM to the pre-trained LM Llama. As another example, Llama-2 is a known pre-trained LM, and Llama-2-Chat is an aligned LM to the pre-trained LM Llama-2. It should be understood that the pre-trained LM and the corresponding aligned LM are not limited to the example LMs; the solution disclosed herein is applicable to any suitable pre-trained LM and corresponding aligned LM.

[0027] Figure 2 Exemplary language problems according to various aspects of this disclosure are illustrated.

[0028] A significant portion of downstream tasks or applications can be adapted to multi-choice settings, where logit-based undetermined quantization (UQ) is tractable. Consider a task whose samples are given by instructions (or questions). x and a set of candidate responses Composition. To evaluate language responses through language modeling. The sample can be Format it as a multiple-choice question. For example... Figure 2 As shown, mapping can be used. Create the question body for multiple choice questions 200. The problem body 200 may include the task description 210, the example in the context 220, and the problem example 230. The problem example 230 may include the problem description 2310 and all candidate responses. 2320. For each candidate Assign option letters 2330, and trigger token 2340 and target generation location 2350 may also be optionally included in the question body 200. Task description 210 and context example 220 are optional and may not be included in the question body 200. If context example 220 is included in the question body 200, ICL is used, where context example 220 provides a demonstration for LM. If context example 220 is not included in the question body 200, zero-shot learning (ZSL) is used, where no context example 220 provides a demonstration for LM. LM 110 or 120 can then estimate the probability of having an option letter on all given tokens. The answer .

[0029] The prediction of LM is expressed as And represent the truth value as Uncertainty calibration checks the accuracy of LM. and confidence level Consistency at the overall level between them. A perfectly calibrated model maintains... Therefore, it has zero expected calibration error (ECE), i.e. To evaluate calibration using ECE in a practice with a finite number of N samples, the model's confidence level can be divided into... M Ten bins of equal size can be used to estimate the ECE in this embodiment, denoted as ECE10. Represented as the first m The index of the samples in each bin can be estimated by weighted average of the differences between confidence and accuracy in each bin: (1)

[0030] Figure 3 Exemplary reliability diagrams are shown according to various aspects of this disclosure.

[0031] Figure 3 The dashed line in the diagram indicates perfect calibration (PC) of the LM, meaning the LM's confidence (CON) and accuracy (ACC) are consistent, with zero ECE. The darker line indicates the reliability of the pre-trained LM (Llama in this example), and the brighter line indicates the reliability of the aligned LM (Vicuna in this example). Figure 3 As shown, although the overall accuracy of the aligned LM (58.63) is higher than that of the pre-trained LM (57.76), the aligned LM is overconfident, as can be seen from the comparison between the PC line and the aligned LM line. That is, the aligned LM is miscalibrated due to its overconfidence in multiple-choice questions based on logistic value evaluation. On the other hand, the large pre-trained LM is well calibrated, as can be seen from the comparison between the PC line and the pre-trained LM line. Because the aligned LM is overconfident in its predictions, the miscalibration of the aligned LM in terms of confidence may hinder its application in many scenarios.

[0032] To find a way to mitigate the miscalibration of aligned LMs, comprehensive experiments were performed on several pairs of pre-trained and aligned LMs, focusing on the differences in calibration and other relevant metrics between the two under multiple selection settings. An exemplary experimental setup is shown below.

[0033] Models: The Llama family, ranging from model size 7B to 70B, is used as the pre-trained LM, while Vicuna and Llama-2-Chat are used as aligned LMs for the pre-trained Llama and Llama-2, respectively.

[0034] Prompt format: All data uses a multiple-choice format. For example... Figure 2 As shown, the input prompts include an optional task description 210, an optional context example 220, and a test example 230. Two formats can be used for the options: "A" and "(A)". In the former case, LM needs to output the option letter directly after "Answer:", while in the latter case, LM will use the trigger token "(" as a prompt and output the option letter following it.

[0035] Evaluation Protocol: Zero-shot learning (ZSL) and few-shot learning in context (ICL) evaluations are performed for each dataset. For the zero-shot setting, the example portion 220 in the context of the cue words is simply omitted. For the few-shot setting, the example portion 220 in the context of the cue words is retained.

[0036] Metrics: The evaluation is based on the output logical value of the LM at a specific target generation location. For example... Figure 2 As shown, for example, the option letter with the highest probability among all options can be used as the prediction of the LM, and its probability across the entire token space can be used as the prediction confidence of the LM. For each task, accuracy and ECE10 are used as the primary metrics. The average prediction confidence of the LM, the sum of the probabilities of all option letters, and the probability of triggering a token (if any) can be tracked to better understand the behavior of the LM.

[0037] Figure 4 The average accuracy of the LM across all seven tasks and all tests, and exemplary experimental results for ECE, are shown out of the box, based on various aspects of this disclosure.

[0038] Out-of-the-box results refer to the results obtained using the confidence scores of the Model Model (LM) without post-processing. Model sizes (MS) include 7B, 13B, 33B, or 70, and LMs include two pre-trained LMs (Llama-1, Llama-2) and two corresponding aligned LMs (Vicuna-v1.3, Llama-2-Chat). In the ICL setting, the pre-trained LMs generally have low ECE, with the largest pre-trained model, Llama-2 70B, achieving the best performance in both accuracy and ECE among all pre-trained and aligned LMs. Regardless of size, all aligned LMs have higher ECEs than their corresponding pre-trained models. Unlike the pre-trained LMs, the aligned LMs show less variation in accuracy and ECE between the ZSL and ICL settings.

[0039] To better understand how calibration varies between pre-trained and aligned LMs, as well as between ZSL and ICL settings, monitor how the prediction confidence of the LMs behaves with these variables. Figure 5 The experimental results for accuracy (ACC), ECE, and average confidence level (AVG CON) under different settings are shown according to various aspects of this disclosure. Figure 5 The upper part corresponds to the selection format "A". Figure 5 The lower part corresponds to the selection format "(A)". For example... Figure 5 As shown in the upper section, for the pre-trained LM under the choice format "A", the sum of probabilities of the option letters (SoCL) is low in the ZSL setting. This indicates that the pre-trained LM is uncertain about the format of its responses, i.e., whether it should directly output the option letters of a multiple-choice question in the first place; this can be called format uncertainty. This uncertainty leads to low prediction confidence, resulting in a high ECE for the pre-trained LM in the ZSL setting. For the ICL setting, the pre-trained LM eliminates format uncertainty by following examples in the context. Notably, the prediction confidence adjusted by ICL is a well-calibrated out-of-the-box prediction confidence, indicating that the pre-trained LM is a calibrated context learner. On the other hand, under the choice format "(A)", the format uncertainty is decomposed into the given trigger token "(", so the difference between the calibration of the model on ZSL and ICL is smaller in this setting. Nevertheless, the model's prediction confidence is still slightly affected by ICL, possibly because examples in the context can provide information beyond the response format of the LM.

[0040] Compared to pre-trained LMs, aligned LMs are overly confident in their out-of-the-box outputs in both ZSL and ICL settings, preferring to directly output option letters to answer multiple-choice questions, i.e., they exhibit relatively lower format uncertainty compared to pre-trained LMs. Applying ICL to aligned LMs can also eliminate format uncertainty, but unlike pre-trained LMs, it has little impact on accuracy and calibration. These results indicate that the alignment process disrupts the well-calibrated prediction distribution of pre-trained LMs and cannot be repaired by ICL during inference.

[0041] To address the miscalibration problem caused by the alignment process, an effective approach is to perform post-calibration of the prediction confidence of the aligned language model. Unlike traditional deep learning models, language models will face multiple tasks simultaneously in real-world deployments, thus requiring a computationally friendly, sample-efficient, and accuracy-preserving calibration method.

[0042] Figure 6 An exemplary process for predicting the confidence level of an aligned LM according to various aspects of this disclosure is shown.

[0043] Verification set of language data The inputs are fed into the pre-trained LM110 and the aligned LM120, where This represents a pair of language questions and answers. For each language question... Each of the pre-trained LM110 and the aligned LM120 generates a confidence distribution for the predicted answer. In other words, the aligned LM 120 generates the confidence distribution of predicted answers. The confidence distribution of the pre-trained LM110 generated predicted answers .by Figure 2 Taking the example question shown, there are four candidate options with four letters each. Predict the confidence distribution of the answer. This represents the individual probabilities of the four candidate options; therefore, in this example, it could be a vector of four elements indicating probabilities.

[0044] The aligned LM 120 includes a post-calibration module 1210. The post-calibration module 1210 can perform temperature scaling to refine the predicted distribution of the aligned LM 120. In one embodiment, the post-calibration module 1210 can adjust the original logic values ​​at the positions of the option letters. Scaling, especially by temperature scaling factor T Original logical value Zoom to .by Figure 2 Taking the problem shown as an example, the original logical value These are four raw numbers corresponding to the four option letters. Then, the probability generation module 1220 processes the scaled logical values. To generate the confidence distribution of predicted answers This includes the corresponding probabilities of the four candidate options. In one embodiment, the probability generation module 1220 can be implemented using a softmax function, i.e. = softmax( ).

[0045] Distance determination module 130 can determine the confidence distribution of the predicted answer of the aligned LM 120. The confidence distribution of predicted answers from pre-trained LM110 The distance between them. In one embodiment, this distance can be the Kullback-Leibler (KL) divergence: In one embodiment, the cumulative distance over the verification set can be obtained at the distance determination module 130: .

[0046] Parameter update module 140 can minimize KL divergence. Update the temperature scaling factor in the direction T .coefficient T The update can be performed using gradient descent, which is a known method for updating parameters.

[0047] Updated temperature scaling factor output from module 140 T Used to update the temperature scaling factor of the post-calibration module 1210 T Then in the next iteration, use the new temperature scaling factor. T Repeat the above process iteratively to minimize the KL divergence. This uses a temperature scaling method to enforce consistency in the prediction distributions between the aligned LM and the pre-trained LM, where the degree of variation in the prediction distribution of the aligned LM relative to the pre-trained LM is achieved by simply using a parameter. T I came here to learn.

[0048] Figure 7 The calibration results of the proposed calibration method are shown in comparison with other baseline methods according to various aspects of this disclosure.

[0049] All calibration methods were tested on an aligned LM Llama-2-Chat 70B, whose corresponding pre-trained LM Llama-2 70B demonstrated the best calibration performance among all tested LMs. Figure 6In the examples, the baseline methods include the out-of-the-box (OOTB) method, the few-sample temperature scaling (TS) method, the kernel density estimation (KDE) method, the TS with constant T=2.5, and the enhanced TS (ETS) method. These five methods are represented by different shades, such as... Figure 7 As shown. The horizontal axis represents different tasks or applications, with testing and seven tasks illustrated. Figure 7 As shown, neither TS nor KDE calibrates the LM well for all tasks with a small number of sample examples. In some tasks (e.g., LogiQA, IMDB), their roles can be complementary, while in others (e.g., OpenbookQA), both are worse than out-of-the-box calibration. The prior overconfidence of aligned LMs leads to the application of a uniform temperature (T==2.5) across all tasks as a strong baseline. However, the optimal temperature can vary significantly across different tasks, resulting in suboptimal solutions. Among these methods, the proposed ETS is the only one that outperforms out-of-the-box calibration on all tasks and calibrates the language model most effectively in the vast majority of scenarios. This demonstrates that learning the predictive distribution of an aligned LM relative to the pre-trained LM by using one parameter for each task is an efficient and powerful post-calibration solution.

[0050] Figure 8 An exemplary process for calibrating a language model (LM) according to various aspects of this disclosure is shown.

[0051] At step 810, the first LM receives a set of language questions and generates a first set of predicted answer confidence distributions corresponding to each of the language questions. For example, the set of language questions could be questions from a validation set.

[0052] At step 820, the second LM receives the set of language questions and generates a second set of predicted answer confidence distributions corresponding to the set of language questions, wherein generating the second set of predicted answer confidence distributions includes calibrating the second set of predicted answer confidence distributions based on a temperature scaling factor.

[0053] At step 830, the distance between the confidence distributions of the first set of predicted answers and the confidence distributions of the second set of predicted answers is determined. According to one embodiment, this distance is the Kullback-Leibler (KL) divergence.

[0054] At step 840, the temperature scaling factor is updated based on the distance.

[0055] According to one embodiment, steps 810 to 840 are performed iteratively to minimize the distance.

[0056] According to one embodiment, the first LM is a pre-trained LM, and the second LM is an aligned LM obtained by fine-tuning the pre-trained LM.

[0057] According to one embodiment, the first LM and the second LM are configured to have an ICL.

[0058] According to one embodiment, fine-tuning the pre-trained LM includes fine-tuning the pre-trained LM using the SFT method and / or the LPF method.

[0059] According to one embodiment, calibrating the second set of predicted answer confidence distributions includes: scaling the original logic value corresponding to each predicted answer confidence distribution in the second set of predicted answer confidence distributions based on a temperature scaling factor; and generating the predicted answer confidence distribution based on the scaled original logic value.

[0060] According to one embodiment, generating a predicted answer confidence distribution based on scaled raw logic values ​​includes: generating the predicted answer confidence distribution by applying a softmax function to the scaled raw logic values.

[0061] According to one embodiment, the set of language questions are multiple-choice questions, and each of the first set of predicted answer confidence distributions and the second set of predicted answer confidence distributions is a probability distribution of the candidate options for the corresponding language question.

[0062] According to one embodiment, the set of language questions are multiple choice questions, and the original logical value corresponding to each predicted answer confidence distribution in the second set of predicted answer confidence distributions is the original logical value at the position of the option letter of the candidate option for the corresponding language question.

[0063] Figure 9 Exemplary processes for performing language tasks according to various aspects of this disclosure are shown.

[0064] At step 910, the LM receives the language problem, where the LM is obtained by using... Figure 8 The calibration method and the calibration method obtained by using various aspects of this disclosure. For example, LM is already used Figure 8 The calibration method and the aligned LM calibrated by using the calibration method according to various aspects of this disclosure.

[0065] At step 920, LM outputs the answer in response to the language question.

[0066] Figure 10An exemplary computing system according to various aspects of this disclosure is illustrated. The computing system 1000 may include at least one processor 1010. The computing system 1000 may also include at least one storage device 1020. The storage device 1020 may store computer-executable instructions, which, when executed, cause the processor 1010 to perform actions according to the combination... Figure 1-9 Any operation of the embodiments described in this disclosure.

[0067] Embodiments of this disclosure may be embodied in a computer-readable medium (e.g., a non-transitory computer-readable medium). A non-transitory computer-readable medium may include instructions that, when executed, cause one or more processors to perform actions according to the combination... Figure 1-9 Any operation of the embodiments described in this disclosure.

[0068] Embodiments of this disclosure can be embodied in a computer program product including computer-executable instructions that, when executed, cause one or more processors to perform according to the combination Figure 1-9 Any operation of the embodiments described in this disclosure.

[0069] It should be understood that all operations in the above methods are merely exemplary, and this disclosure is not limited to any operation in these methods or the order of such operations, and should cover all other equivalents under the same or similar concepts.

[0070] It should also be understood that all modules in the aforementioned apparatus can be implemented using various methods. These modules can be implemented as hardware, software, or a combination thereof. Furthermore, any of these modules can be further functionally divided into sub-modules or combined together.

[0071] The foregoing description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to limit them to the aspects shown herein. All structural and functional equivalents of elements throughout the various aspects described in this disclosure that are known to or will later become apparent to those skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims.

Claims

1. A computer-implemented method for calibrating a language model (LM), comprising: The first LM receives a set of language questions and generates a first set of predicted answer confidence distributions corresponding to the set of language questions. The second LM receives the set of language questions and generates a second set of predicted answer confidence distributions corresponding to the set of language questions, wherein generating the second set of predicted answer confidence distributions includes calibrating the second set of predicted answer confidence distributions based on a temperature scaling factor; Determine the distance between the confidence distributions of the first set of predicted answers and the confidence distributions of the second set of predicted answers; and The temperature scaling factor is updated based on the distance.

2. The method according to claim 1, wherein, Iteratively perform reception and generation by the first LM, reception and generation by the second LM, determine the distance, and update the temperature scaling factor to minimize the distance.

3. The method according to claim 1, wherein, The first LM is a pre-trained LM, and the second LM is an aligned LM obtained by fine-tuning the pre-trained LM.

4. The method according to claim 3, wherein, The first LM and the second LM are configured to have in-context learning (ICL).

5. The method according to claim 3, wherein, The fine-tuning of the pre-trained LM includes fine-tuning the pre-trained LM through supervised fine-tuning (SFT) methods and / or learning from pairwise feedback (LPF) methods.

6. The method according to claim 1, wherein, The calibration of the confidence distribution of the second set of predicted answers includes: The original logical values ​​corresponding to each predicted answer confidence distribution in the second set of predicted answer confidence distributions are scaled based on the temperature scaling factor; and The confidence distribution of the predicted answer is generated based on the scaled original logical values.

7. The method according to claim 6, wherein, The process of generating the predicted answer confidence distribution based on the scaled original logistic value includes: The confidence distribution of the predicted answer is generated by applying the softmax function to the scaled original logical values.

8. The method according to claim 1, wherein, The distance is the Kullback-Leibler (KL) divergence.

9. The method according to claim 1, wherein, The set of language questions are multiple-choice questions, and each of the first set of predicted answer confidence distributions and the second set of predicted answer confidence distributions is a probability distribution of the candidate options for the corresponding language question.

10. The method according to claim 6, wherein, The set of language questions are multiple-choice questions, and the original logical value corresponding to each predicted answer confidence distribution in the second set of predicted answer confidence distributions is the original logical value at the position of the option letter of the candidate option for the corresponding language question.

11. A computer-implemented method for performing language tasks, comprising: The LM receiver language problem is obtained by using the calibration method according to any one of claims 1-10; The LM outputs an answer in response to the language question.

12. A computer system, comprising: One or more processors; as well as One or more storage devices storing computer-executable instructions that, when executed, cause the one or more processors to perform the operation of the method according to any one of claims 1-11.

13. One or more computer-readable storage media storing computer-executable instructions that, when executed, cause one or more processors to perform the operation of the method according to any one of claims 1-11.

Images