A question and answer prediction method, device and equipment based on a large language model and a storage medium

Abstract

Embodiments of the present application disclose a kind of based on the method, device and storage medium of question and answer prediction of large language model, wherein, method includes: obtaining target open query and corresponding calibration dataset;Calibration query is input into the target question and answer model that has been pre-trained, corresponding candidate answer set is obtained, reliability analysis is carried out based on candidate answer in candidate answer set, and target calibration threshold is determined;Target open query is input into target question and answer model, corresponding selectable answer set is obtained, and selectable answer set is filtered based on target calibration threshold, at least one target answer corresponding to target open query is determined.The technical scheme of the present application embodiment determines target calibration threshold by carrying out reliability analysis to candidate answer set, and then filters the answer corresponding to target open query based on target calibration threshold, which can improve the credibility, explainability and candidate set efficiency of the result in open generation scenario.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of natural language processing technology, and in particular to a question-answering prediction method, apparatus, device and storage medium based on a large language model. Background Technology

[0002] Large language models possess powerful open-ended text generation capabilities and have been widely applied in scenarios such as question answering, reasoning, dialogue, information extraction, and decision support. Traditionally, the model's most likely response is used as a single-point prediction output. However, in open-ended generation tasks, single-point predictions do not fully reflect the true generation capabilities of large language models: the highest-ranked answer may contain factual errors, illusions, or semantic biases, while other candidate answers obtained through repeated sampling may still contain correct or acceptable responses.

[0003] Existing uncertainty quantification methods typically determine the credibility of a single answer using metrics such as entropy, lexical probability, model self-evaluation, consistency, or semantic uncertainty. While these methods can provide risk signals, they are mostly heuristic judgments and struggle to provide rigorous coverage or risk control guarantees under limited sample conditions. Especially in high-risk scenarios such as medical, financial, legal, and scientific question answering, simply outputting an uncalibrated answer or a confidence score lacking statistical guarantees is insufficient to meet reliable deployment requirements.

[0004] Conformal prediction and conformal risk control can provide distribution-independent statistical guarantees for finite samples under the commutativity assumption, but common methods are mainly geared towards classification or closed candidate spaces, assuming the true labels reside in an enumerable set of labels. For open-ended language generation, the output space is approximately infinite, and the candidate answer set obtained from a finite number of samplings may not contain any candidate answers semantically consistent with the standard answer. When the candidate pool itself does not contain the correct answer, no matter how it is subsequently sorted, filtered, or thresholded, it is impossible to construct a predicted answer set containing the correct answer. Summary of the Invention

[0005] This invention provides a question-answering prediction method, apparatus, device, and storage medium based on a large language model. The technical solution of this invention solves the problems of single-point output being susceptible to illusions and the candidate answer set possibly not containing the correct answer in the open query generation of existing large language models. It can first repeatedly sample the calibration query to obtain a candidate answer set, determine the target calibration threshold by performing reliability analysis on the candidate answer set, and then filter the answers corresponding to the target open query based on the target calibration threshold, which can improve the credibility, interpretability, and candidate set efficiency of the results in open generation scenarios.

[0006] In a first aspect, embodiments of the present invention provide a question-answering prediction method based on a large language model, the method comprising: Obtain a target open-ended query and its corresponding calibration dataset; wherein the calibration dataset includes at least one calibration query and a reference answer corresponding to the calibration query; wherein the calibration query and the target open-ended query belong to the same technical field; input the calibration query into a pre-trained target question-answering model to obtain a corresponding set of candidate answers, perform reliability analysis on the candidate answers in the set of candidate answers, and determine a target calibration threshold; input the target open-ended query into the target question-answering model to obtain a corresponding set of optional answers, and filter the set of optional answers based on the target calibration threshold to determine at least one target answer corresponding to the target open-ended query.

[0007] Secondly, embodiments of the present invention provide a question-answering prediction device based on a large language model, the device comprising: A data acquisition module is used to acquire a target open-ended query and a corresponding calibration dataset; wherein the calibration dataset includes at least one calibration query and a reference answer corresponding to the calibration query; wherein the calibration query and the target open-ended query belong to the same technical field; a calibration threshold determination module is used to input the calibration query into a pre-trained target question-answering model to obtain a corresponding set of candidate answers, and perform reliability analysis on the candidate answers in the set of candidate answers to determine a target calibration threshold; an answer determination module is used to input the target open-ended query into the target question-answering model to obtain a corresponding set of optional answers, and filter the set of optional answers based on the target calibration threshold to determine at least one target answer corresponding to the target open-ended query.

[0008] Thirdly, embodiments of the present invention provide a computer device, the computer device comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the question-answering prediction method based on a large language model as described in any embodiment.

[0009] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the question-answering prediction method based on a large language model as described in any embodiment.

[0010] The technical solution provided by this invention involves obtaining a target open-ended query and a corresponding calibration dataset. The calibration dataset includes at least one calibration query and a corresponding reference answer. The calibration query and the target open-ended query belong to the same technical field. The calibration query is input into a pre-trained target question-answering model to obtain a set of candidate answers. Reliability analysis is performed on the candidate answers in the set to determine a target calibration threshold. The target open-ended query is then input into the target question-answering model to obtain a set of optional answers. Based on the target calibration threshold, the optional answers are filtered to determine at least one target answer corresponding to the target open-ended query. This invention addresses the problems of existing large language models where single-point output is susceptible to illusions and the candidate answer set may not contain correct answers. It first repeatedly samples the calibration query to obtain a set of candidate answers, then performs reliability analysis on the candidate answer set to determine the target calibration threshold, and finally filters the answers corresponding to the target open-ended query based on the target calibration threshold. This improves the credibility, interpretability, and candidate set efficiency of results in open-ended generation scenarios. Attached Figure Description

[0011] Figure 1 This is a flowchart of a question-answering prediction method based on a large language model provided in an embodiment of the present invention; Figure 2 This is a flowchart of another question-answering prediction method based on a large language model provided in an embodiment of the present invention; Figure 3 This is a flowchart of determining the reliability score corresponding to a candidate answer, provided by an embodiment of the present invention; Figure 4 This is a flowchart of determining a calibration threshold provided in an embodiment of the present invention; Figure 5 This is a flowchart of determining the corresponding answer for a target query, provided by an embodiment of the present invention; Figure 6 This is a flowchart of a question-answering prediction based on a large language model provided by an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a question-answering prediction device based on a large language model provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The acquisition, storage, use, and processing of data in the technical solutions of the embodiments of the present invention all comply with the relevant provisions of national laws and regulations.

[0013] Figure 1 This is a flowchart of a question-answering prediction method based on a large language model provided by an embodiment of the present invention. The embodiment of the present invention can be applied to scenarios of answering open-ended query questions. The method can be executed by a question-answering prediction device based on a large language model, which can be implemented by software and / or hardware.

[0014] like Figure 1 As shown, the question-answering prediction method based on a large language model includes the following steps: S110. Obtain the target open query and the corresponding calibration dataset.

[0015] The target open-ended query can be an open-ended query question that requires a solution. Specifically, the answer to the target open-ended query is not fixed; the technical solution of this embodiment can filter out the more reliable answers from the available answers to the target open-ended query. The calibration dataset can be a dataset used for domain calibration of the target open-ended query. The calibration dataset includes at least one calibration query and a reference answer corresponding to the calibration query. The calibration query and the target open-ended query belong to the same technical field. The calibration query can also be an open-ended query question. Subsequent analysis of the calibration dataset can determine the threshold parameters used for answer filtering within the technical field corresponding to the target open-ended query. Then, the available answers to the target open-ended query are filtered based on the determined threshold parameters.

[0016] S120. Input the calibration query into the pre-trained target question-answering model to obtain the corresponding candidate answer set. Perform reliability analysis based on the candidate answers in the candidate answer set to determine the target calibration threshold.

[0017] The target question-answering model can be a large language model used for answering open-ended questions. The candidate answer set can be the set of answers obtained after the target question-answering model answers the calibration query. Specifically, each calibration query can be input into a pre-trained target question-answering model to obtain a candidate answer set corresponding to each calibration query. Furthermore, the target calibration threshold can be a reference threshold used to determine the reliability of answers to the target open-ended query. The candidate answer set can contain multiple candidate answers corresponding to the calibration query. Reliability analysis can be performed on each candidate answer in the candidate answer set, and the target calibration threshold can be selected from the multiple candidate calibration thresholds based on the reliability analysis results.

[0018] S130. Input the target open query into the target question answering model to obtain the corresponding set of optional answers, and filter the set of optional answers based on the target calibration threshold to determine at least one target answer corresponding to the target open query.

[0019] The optional answer set can be the set of answers obtained after answering the target open-ended query based on the target question-answering model. The optional answer set can contain multiple optional answers corresponding to the target open-ended query. The target answer can be the final answer obtained from the target open-ended query. Specifically, reliability analysis can be performed on the optional answers in the optional answer set to determine the reliability score of each optional answer. Then, the reliability scores of the optional answers are evaluated based on a target calibration threshold, thereby determining at least one target answer corresponding to the target open-ended query from the optional answer set.

[0020] The technical solution provided by this invention involves obtaining a target open-ended query and a corresponding calibration dataset. The calibration dataset includes at least one calibration query and a corresponding reference answer. The calibration query and the target open-ended query belong to the same technical field. The calibration query is input into a pre-trained target question-answering model to obtain a set of candidate answers. Reliability analysis is performed on the candidate answers in the set to determine a target calibration threshold. The target open-ended query is then input into the target question-answering model to obtain a set of optional answers. Based on the target calibration threshold, the optional answers are filtered to determine at least one target answer corresponding to the target open-ended query. This invention addresses the problems of existing large language models where single-point output is susceptible to illusions and the candidate answer set may not contain correct answers. It first repeatedly samples the calibration query to obtain a set of candidate answers, then performs reliability analysis on the candidate answer set to determine the target calibration threshold, and finally filters the answers corresponding to the target open-ended query based on the target calibration threshold. This improves the credibility, interpretability, and candidate set efficiency of results in open-ended generation scenarios.

[0021] Figure 2 This is a flowchart of another question-answering prediction method based on a large language model provided by an embodiment of the present invention. The embodiments of the present invention can be applied to scenarios of answering open-ended query questions. Based on the above embodiments, this embodiment further explains how to determine the target calibration threshold by performing reliability analysis based on candidate answers in the candidate answer set. The device can be implemented by software and / or hardware and integrated into a computer device with application development capabilities.

[0022] like Figure 2 As shown, the question-answering prediction method based on a large language model includes the following steps: S210. Obtain the target open query and the corresponding calibration dataset.

[0023] The target open-ended query can be an open-ended query question that requires a solution. Specifically, the answer to the target open-ended query is not fixed; the technical solution of this embodiment can filter out the more reliable answers from the available answers to the target open-ended query. The calibration dataset can be a dataset used for domain calibration of the target open-ended query. The calibration dataset includes at least one calibration query and a reference answer corresponding to the calibration query. The calibration query and the target open-ended query belong to the same technical field. The calibration query can also be an open-ended query question. Subsequent analysis of the calibration dataset can determine the threshold parameters used for answer filtering within the technical field corresponding to the target open-ended query. Then, the available answers to the target open-ended query are filtered based on the determined threshold parameters.

[0024] For example, a calibration dataset can be obtained. ,in This represents the i-th calibration query. This represents the corresponding reference answer; it defines the large language model (target question-answering model) G, the candidate sampling budget K, and the user-specified target risk value (i.e., the user's target risk value). The semantic alignment function A and the reliability evaluation formula F are used for the target query to be processed (i.e., the target open query). The true answer (i.e., the target answer). It is unknown during the prediction phase.

[0025] S220. Input the calibration query into the pre-trained target question-answering model to obtain the corresponding candidate answer set. Evaluate the reliability of each candidate answer based on the preset reliability evaluation formula and determine the reliability score corresponding to each candidate answer.

[0026] The target question-answering model can be a large language model used for answering open-ended questions. The candidate answer set can be the set of answers obtained after the target question-answering model answers the calibration query. Specifically, each calibration query can be input into a pre-trained target question-answering model to obtain a candidate answer set corresponding to each calibration query. The candidate answer set can contain multiple candidate answers corresponding to the calibration query.

[0027] For example, for each calibration query The large language model G is invoked to perform K open-ended generation operations to form a set of candidate answers. For the target query Similarly, a set of candidate answers is generated. Sampling methods may include temperature sampling, nuclear sampling, random sampling, beam search sampling, or a combination thereof.

[0028] Furthermore, the preset reliability evaluation formula can be a pre-defined formula used to evaluate the reliability of the answer. The reliability score can be a parameter used to represent the reliability of the candidate answer. Specifically, the reliability of each candidate answer can be evaluated separately based on the preset reliability evaluation formula, and the reliability score corresponding to each candidate answer can be determined separately.

[0029] Optionally, before performing reliability analysis based on candidate answers in the candidate answer set, the following may also be included: Semantic alignment analysis is performed on the candidate answer set corresponding to each calibration query and the reference answer corresponding to the calibration query. The minimum achievable risk value is determined based on the semantic alignment analysis results. If the minimum achievable risk value is greater than the user's target risk value, the effective risk value is adjusted so that the adjusted effective risk value is not less than the minimum achievable risk value.

[0030] The minimum achievable risk value can be the risk value corresponding to the inability to correctly answer a candidate query. Specifically, semantic alignment analysis can be performed on the candidate answer set corresponding to each calibration query and the reference answer corresponding to the calibration query, and the minimum achievable risk value can be determined based on the semantic alignment analysis results. The user's target risk value can be the risk value that the user can accept for the inability to correctly answer a target open query. Specifically, the user's target risk value can be selected by the user when inputting the target open query. The effective risk value can be the risk threshold used in determining the target calibration threshold. If the minimum achievable risk value is greater than the user's target risk value, it indicates that the user's risk control needs cannot be met under the current large language model and semantic judgment criteria. In this case, the effective risk value can be adjusted so that the adjusted effective risk value is not less than the minimum achievable risk value, reducing the risk of the target open query not being answered correctly. In addition, the user can be prompted to adjust their target risk value appropriately so that the minimum achievable risk value is not greater than the adjusted user target risk value.

[0031] Optionally, semantic alignment analysis is performed on the candidate answer set corresponding to each calibration query and the reference answer corresponding to the calibration query, and the minimum achievable risk value is determined based on the semantic alignment analysis results. This includes: for each candidate answer set, performing semantic similarity analysis on each candidate answer in the candidate answer set and the reference answer corresponding to the calibration query, and determining the candidate similarity corresponding to each candidate answer; determining the failure indicator corresponding to the candidate answer set based on the preset semantic alignment threshold and candidate similarity, and substituting the failure indicator of the entire candidate answer set into the achievable risk determination formula to obtain the minimum achievable risk value.

[0032] Here, candidate similarity can be the semantic similarity between a candidate answer and a reference answer. Failure indicator can be a parameter representing the degree of semantic deviation between the candidate answer set and its corresponding reference answer. Specifically, it can be determined whether the candidate answer set can semantically align with the corresponding reference answer based on candidate similarity, and the failure indicator corresponding to the candidate answer set can be determined based on the determination result. For example, if all candidate answers in the candidate answer set cannot semantically align with the reference answer (the candidate similarity corresponding to all candidate answers is less than the alignment determination threshold), the failure indicator is determined to be equal to 1; otherwise, the failure indicator is equal to 0. Furthermore, the failure indicators of the entire candidate answer set can be substituted into the achievable risk determination formula (used to determine the minimum achievable risk value) to determine the minimum achievable risk value.

[0033] For example, a semantic alignment function can be defined. Semantic alignment function Used to determine candidate answers Is it consistent with the reference answer? They are semantically equivalent or acceptable in the context of the task. When the candidate answer is semantically consistent with the reference answer... =1; otherwise =0. The semantic alignment function can be implemented by sentence vector similarity, cross-encoder similarity, natural language inference models, exact matching rules, task scorers, or manual annotation. In one embodiment, when the semantic alignment function A is implemented by the semantic similarity function... During implementation, a preset semantic alignment threshold is used. ;like ,but ,otherwise =0.

[0034] Define a finite sampling failure indicator for each calibration sample. When the set of candidate answers None of the candidate answers can be compared with the reference answer. During semantic alignment, =1; otherwise =0. Therefore, the minimum realizable risk value under a given sampling budget K is estimated. .

[0035] (1) (2) in, This indicates the number of calibration queries in the calibration dataset. This indicates the unavoidable risk that even if all candidate answers from the candidate answer set are included in the predicted answer set, the correct semantic answer still cannot be included. This applies when the user's target risk... Below This indicates that, under the current large language model, sampling budget K, and semantic judgment criteria, the nominal risk... Unfeasible; the risk controller outputs an infeasibility warning, or the effective risk value may be reduced. Adjusted to no less than The value of .

[0036] Optionally, reliability evaluation is performed on each candidate answer based on a preset reliability evaluation formula, and a reliability score is determined for each candidate answer. This includes: determining a cross-sample consistency parameter based on the semantic similarity between candidate answers in the candidate answer set; determining a self-uncertainty parameter based on the correlation analysis between the candidate answer and the calibration query; performing semantic clustering on the candidate answers in the candidate answer set, and determining the semantic consensus strength based on the data volume comparison results between semantic clusters; and determining the reliability score corresponding to the candidate answer based on the cross-sample consistency parameter, the self-uncertainty parameter, and the semantic consensus strength.

[0037] The cross-sample consistency parameter can be a parameter used to represent the degree of semantic consistency between candidate answers. Specifically, the semantic similarity between candidate answers in the candidate answer set can be determined first. For each candidate answer, the semantic similarity between that candidate answer and other candidate answers is substituted into the cross-sample consistency determination formula to obtain the cross-sample consistency parameter corresponding to that candidate answer.

[0038] Furthermore, the self-uncertainty parameter can be a parameter used to represent the degree of association between candidate answers and calibration queries. Specifically, association analysis can be performed based on candidate answers and calibration queries, and the corresponding self-uncertainty parameter can be determined based on the degree of association between candidate answers and calibration queries. Semantic consensus strength can be a parameter used to represent the degree of semantic clustering among candidate answers. Specifically, semantic clustering can be performed on candidate answers in the candidate answer set to obtain multiple semantic clusters, and the corresponding semantic consensus strength can be determined based on the number of candidate answers in the semantic cluster to which a candidate answer belongs.

[0039] Optionally, based on the semantic similarity between candidate answers in the candidate answer set, a cross-sample consistency parameter is determined, including: for each current candidate answer, determining the semantic similarity between the current candidate answer and each other candidate answer in the candidate answer set, and using the average of all semantic similarities as the cross-sample consistency parameter corresponding to the current candidate answer.

[0040] Optionally, semantic clustering is performed on the candidate answers in the candidate answer set, and the semantic consensus strength is determined based on the comparison of data volume between semantic clusters. This includes: performing semantic clustering on the candidate answers in the candidate answer set to obtain multiple semantic clusters; for each current candidate answer, comparing the data volume of the semantic cluster where the current candidate answer is located with that of the largest semantic cluster, and using the ratio as the semantic consensus strength of the current candidate answer.

[0041] Optionally, the reliability score corresponding to the candidate answer is determined based on the cross-sample consistency parameter, the self-uncertainty parameter and the semantic consensus strength, including: normalizing the cross-sample consistency parameter and the self-uncertainty parameter to obtain the target consistency parameter and the target uncertainty parameter; weighting and summing the target consistency parameter and the target uncertainty parameter to obtain the basic quality score, and using the product of the basic quality score and the semantic consensus strength as the reliability score.

[0042] The target consistency parameter can be obtained by normalizing the cross-sample consistency parameter. The target uncertainty parameter can be obtained by normalizing the self-uncertainty parameter. The basic quality score can be the reliability score corresponding to the candidate answer, determined comprehensively from the perspectives of semantic similarity and query relevance.

[0043] For example, Figure 3 This is a flowchart illustrating how to determine the reliability score corresponding to a candidate answer, as provided in an embodiment of the present invention. Figure 3 As shown, the process for determining the reliability score corresponding to a candidate answer includes the following steps: Calculate the multi-perspective reliability score for the candidate answer set. candidate answers This invention integrates uncertainty, cross-sample consistency, and semantic consensus strength to obtain a reliability score. A higher reliability score indicates that the candidate answer is more credible and will be given priority in the predicted answer set.

[0044] First, construct pairwise semantic similarity matrices within the same set of candidate answers. And calculate candidate answers Average similarity with other candidate answers To characterize cross-sample consistency (That is, the cross-sample consistency parameter).

[0045] (3) (4) Secondly, the self-uncertainty signal of candidate answers is calculated using word-level probability, attention-related uncertainty index, negative log-likelihood, entropy, or TokenSAR index. (That is, the self-uncertainty parameter), and through monotonic transformation, the larger the value, the more reliable the candidate answer.

[0046] (5) Next, z-score normalization is performed on the self-uncertainty signal and cross-sample consistency within the same candidate answer set, yielding... and According to weight and Calculate the basic quality score .

[0047] (6) Finally, semantic clustering is performed on the candidate answers. If the candidate answers... The size of the semantic cluster is The maximum semantic cluster size is Then the consensus strength The definition is as follows, where The adjustment coefficient is greater than 0.

[0048] (7) The final reliability score of the candidate answer is the product of the basic quality score and the semantic consensus strength: (8) S230. For each candidate calibration threshold, the candidate answer set is filtered based on the candidate calibration threshold and reliability score to obtain the predicted answer set corresponding to the candidate calibration threshold.

[0049] The candidate calibration threshold can be a reference threshold that may be used to determine the reliability of the answer. Specifically, there can be multiple candidate calibration thresholds, each of which can be set manually. Subsequent analysis of the candidate standard thresholds allows for the selection of a reference threshold for determining the reliability of the answer to the target open-ended query. Furthermore, the predicted answer set can be the answer set obtained after filtering the candidate answer set based on the candidate calibration threshold. In this embodiment of the invention, the candidate calibration threshold is represented as... It can retain candidate answers with a reliability score of not less than 1. The candidate answers are used to obtain the predicted answer set.

[0050] For example, for any candidate calibration threshold ∈[0,1], retain a reliability score of not less than 1- The candidate answers were used to obtain calibration samples. The set of predicted answers : (9) In this parameterization method, The larger the value, the higher the screening threshold (1-). The lower the value, the more candidate answers the predicted answer set contains.

[0051] S240. Determine the empirical loss of coverage for each set of predicted answers, and use the minimum candidate calibration threshold that satisfies the risk control condition for the empirical loss of coverage as the target calibration threshold.

[0052] The empirical coverage loss can be derived from the loss function value representing the accuracy of the predicted answer set. Specifically, the difference between each predicted answer set and its corresponding reference answer can be analyzed based on the corresponding empirical coverage loss function to determine the corresponding empirical coverage loss. Optionally, the empirical coverage loss for each predicted answer set can be determined separately, including: for each predicted answer set, determining the set-level coverage loss relative to each reference answer within the predicted answer set, and using the average of all set-level coverage losses as the empirical coverage loss for the predicted answer set.

[0053] The empirical loss of coverage can be a parameter representing the degree of difference between the predicted answer set and the reference answer. Specifically, it can be determined whether there are candidate answers in the predicted answer set that are semantically consistent with the reference answer, and then the value of the empirical loss of coverage can be determined. For example, if there are no candidate answers in the predicted answer set that are semantically consistent with the reference answer, the empirical loss of coverage is equal to 1; if there are candidate answers in the predicted answer set that are semantically consistent with the reference answer, the empirical loss of coverage is equal to 0.

[0054] For example, Figure 4 This is a flowchart illustrating how to determine a calibration threshold, as provided in an embodiment of the present invention. Figure 4 As shown, the process for determining the calibration threshold includes the following steps: Define a set-level loss of coverage based on the set of predicted answers. : (10) When there are no candidate answers in the predicted answer set that are semantically consistent with the reference answer. =1; otherwise =0. With As the set of predicted answers increases, the set of predicted answers monotonically expands, while the loss due to set-level miscovery does not increase monotonically.

[0055] Furthermore, the empirical coverage loss on the calibration set can be calculated. And select the minimum value that satisfies the finite sample risk control condition. As the calibration threshold (i.e., the target calibration threshold) .

[0056] (11) (12) in This is a valid risk value. In one embodiment, when And when equation (12) has a feasible solution, ;when If equation (12) has no solution, the risk controller outputs an infeasibility warning. Threshold search can be achieved through discrete grid search, binary search, or precise search sorted by the reliability score of candidate answers.

[0057] Furthermore, the risk control conditions can be used to determine the risk of empirical coverage loss. Specifically, the method for determining the risk control conditions includes: substituting each empirical coverage loss into a preset risk control function to obtain a candidate risk control function corresponding to each candidate calibration threshold; and confirming that the candidate calibration threshold satisfies the risk control conditions if the risk control function has a feasible solution that is not greater than the effective risk value and the lowest achievable risk value is not greater than the user's target risk value.

[0058] Furthermore, the target calibration threshold can be a reference threshold used to determine the reliability of the answer to the target open query. Specifically, it can be determined whether the empirical loss of coverage corresponding to each candidate calibration threshold meets the risk control conditions, and the smallest candidate calibration threshold whose empirical loss of coverage meets the risk control conditions can be used as the target calibration threshold.

[0059] S250. Input the target open query into the target question answering model to obtain the corresponding set of optional answers, and filter the set of optional answers based on the target calibration threshold to determine at least one target answer corresponding to the target open query.

[0060] The optional answer set can be the set of answers obtained after answering the target open-ended query based on the target question-answering model. The optional answer set can contain multiple optional answers corresponding to the target open-ended query. The target answer can be the final answer obtained from answering the target open-ended query. Specifically, reliability analysis can be performed on the optional answers in the optional answer set to determine the reliability score of each optional answer (the steps for determining the reliability score of optional answers are the same as those for determining the reliability score of candidate answers). Then, the reliability scores of the optional answers are evaluated based on the target calibration threshold, thereby determining at least one target answer corresponding to the target open-ended query from the optional answer set.

[0061] For example, Figure 5 This is a flowchart illustrating how to determine the answer to a target query, as provided in an embodiment of the present invention. For example... Figure 5 As shown, the process of determining the answer to the target query includes the following steps: For target query Generate a set of candidate answers Calculate the reliability score for each candidate answer, and based on... Output target prediction answer set .

[0062] (13) Under the condition that the calibration sample and the target query are commutative, when the user-specified risk value is in the feasible region and the calibration threshold satisfies equation (12), the target prediction answer set has the following statistical coverage guarantee: (14) when = When the above formula is used, it means that the target predicted answer set has at least 1- The probability includes a target answer that is semantically consistent with the true answer.

[0063] Optionally, to reduce semantic redundancy among candidate answers, semantic deduplication can be performed after threshold filtering. Specifically, the semantic similarity between candidate answers within the predicted answer set is calculated, and when the similarity is higher than the deduplication threshold, semantic deduplication is performed. When these are combined into the same semantic cluster, the answer with the highest reliability score, the highest generation probability, or the most concise text is retained as the representative answer.

[0064] Furthermore, the reliability evaluation formula F is a pluggable module. Besides TokenSAR, other sources of uncertainty or reliability can include the average negative log-likelihood of terms, sequence entropy, semantic entropy, model self-evaluation score, multi-model voting consistency, or external retrieval evidence support. Changing the reliability evaluation formula primarily affects the compactness of the predicted answer set, without altering the finite-sample risk control mechanism based on the calibration set.

[0065] Furthermore, the sampling budget K can be adaptively set according to application cost and risk requirements. If If the threshold is too high, the feasible risk area can be expanded by increasing K, improving the capabilities of large language models, optimizing prompt words, introducing search enhancement generation, or lowering the overly strict semantic alignment threshold.

[0066] For example, in order to better understand the technical solution provided by the present invention, specific embodiments are described below. Figure 6 This is a flowchart illustrating a question-answering prediction process based on a large language model, as provided in an embodiment of the present invention. Figure 6 As shown: The workflow for question-answering prediction based on a large language model includes the following steps: Example 1: Set value prediction in open-ended question answering tasks Step 1: Prepare calibration data. Select an open-ended question-and-answer dataset and construct several datasets based on queries. and reference answers The calibration samples are composed of semantic alignment thresholds set according to application requirements. Sampling budget K and target risk value .in, This represents the semantic similarity threshold between the candidate answer and the reference answer, used to determine whether a candidate answer is acceptable when implementing the semantic alignment function A using a semantic similarity model. For example, in one embodiment, K is set to 20, the generation temperature is set to 1.0, and the kernel sampling top-p is set to 0.9. The value can be set to 0.5, 0.6, or 0.7 depending on the task's severity.

[0067] Step 2: Candidate Answer Generation. For each calibration query... The system repeatedly generates K candidate answers using a large language model. During generation, system prompts can be fixed, or retrieved background information can be added to the prompts to make the candidate answers more relevant to the task scenario. The same sampling process is performed on the target query.

[0068] Step 3: Semantic Alignment Determination. During the calibration phase, the semantic similarity between candidate answers and reference answers is calculated using a sentence vector similarity model or a cross-encoder model, denoted as... When the semantic similarity is not lower than If the candidate answer is deemed acceptable, then for tasks requiring strict factual consistency, a natural language inference model can be used to determine whether the candidate answer and the reference answer satisfy a two-way entailment relationship.

[0069] Step 4: Minimum Realizable Risk Estimate. Iterate through the candidate answer set for each calibration query. If no acceptable answer exists, then... Set it to 1, otherwise set it to 0. Calculate according to equation (2). .like A large value indicates that the current model or sampling budget cannot cover enough correct candidate answers, and the risk controller prompts the user to increase the sampling budget K or adjust the model.

[0070] Step 5: Reliability Score Calculation. For each candidate answer, calculate the self-uncertainty signal using TokenSAR, lexical probability, or entropy metrics; calculate the average consensus score using pairwise semantic similarity between candidate answers; merge semantically equivalent candidate answers into consensus clusters using a natural language inference model or similarity threshold, and calculate the consensus strength based on the cluster size. Combine these three perspectives to obtain the final reliability score F.

[0071] Step 6, Threshold Calibration. Perform a grid search on λ from 0 to 1, with a step size of 0.01. Each Corresponding to a set of predicted answers And calculate the set-level loss of coverage according to equation (10). Select the minimum value that satisfies equation (12). As Because the minimum feasible option is chosen. The resulting set of predicted answers is kept as compact as possible while satisfying risk constraints.

[0072] Step 7, Target Prediction. For target queries... Generate K candidate answers and calculate reliability scores. Select answers with scores no lower than 1- Candidate answers are included in the target predicted answer set. If there are semantically highly repetitive answers in the predicted answer set, a deduplication threshold is used. These are then merged, retaining representative answers. Finally, the user is output a set of candidate answers, a reliability score for each candidate answer, and an effective risk value. And feasibility suggestions.

[0073] Example 2: Risk Warnings in Q&A for High-Risk Areas In medical or financial Q&A scenarios, users can set a lower target risk value. To achieve a high coverage rate. If calculated based on the calibration set... Risk exceeding user target If this happens, the system will not directly provide a nominal coverage guarantee, but will instead indicate that "with the current sampling budget and model capabilities, this risk value is unattainable." The system can then automatically increase K for resampling, or adjust the effective risk value to... And indicate the adjustment in the output.

[0074] Through the above method, this invention not only outputs candidate answers but also a judgment on the feasibility of coverage guarantees, thereby avoiding misleading users in high-risk applications. Compared to providing only one answer or only the model confidence level, this invention can provide a more transparent risk boundary and a more robust set of alternative answers.

[0075] Statistical guarantee statement Let the enhancement tuples corresponding to the calibration sample and the target sample be defined. It satisfies commutativity. For any fixed... Collection-level loss of coverage It is a measurable function of the enhanced tuple, therefore the loss sequence also satisfies commutativity. According to the learning-test calibration condition, It is selected as the minimum threshold that makes the empirical loss of the calibration set satisfy the upper bound of the finite samples.

[0076] because Values ​​are in [0,1] and about Monotonic and non-increasing, calibration conditions Ensure that the expected loss of coverage of the target sample does not exceed Therefore, the probability that the target predicted answer set contains at least one semantically correct candidate answer is no less than 1- .when Equivalent to user-specified risk At that time, the user-specified coverage guarantee is obtained.

[0077] Alternative implementation methods In other implementations, the semantic alignment function A can be implemented using manual review, domain knowledge base comparison, structured answer parsing, numerical answer error tolerance, code test case execution results, or an external fact checker. As long as the function can output a binary judgment of whether the candidate answer is acceptable during the calibration phase, it can be integrated into the calibration process of this invention.

[0078] In other implementations, the reliability score F may not use TokenSAR, but rather any scoring function capable of ranking candidate answers, such as generation probability, negative semantic entropy, retrieval evidence support, model self-evaluation score, multi-model consistency score, or a weighted combination of the above scores. A threshold calibration mechanism can conformally preserve this scoring function, thereby transforming the score into a statistically significant set of predicted answers.

[0079] In other implementations, the threshold Instead of a fixed-step grid search, precise enumeration can be achieved using pre-sorted candidate scores. Different calibration sets and different... This allows for deployment across different scenarios.

[0080] The beneficial effects of the technical solution provided by this invention are as follows: (1) This invention extends single-point prediction to set-value prediction, which can utilize the large language model to repeatedly sample existing valid candidate answers, avoiding the direct determination that the model is unusable due to the error of the highest-ranked answer.

[0081] (2) This invention explicitly introduces a minimum achievable risk value. This characterizes the inescapable risk when there is no correct answer in the finite sampling candidate pool, thus avoiding making false coverage commitments under infeasible risk values.

[0082] (3) This invention provides a limited sample statistical coverage guarantee for the target predicted answer set under the condition of commutativity by learning-test threshold calibration, so that the probability of the set containing semantically correct answers meets the risk constraints specified by the user.

[0083] (4) The present invention integrates uncertainty, cross-sample consistency and semantic consensus strength to calculate reliability score, making the candidate answer ranking more stable and reducing the average size of the predicted answer set while satisfying the coverage guarantee.

[0084] (5) This invention is decoupled from specific large language models and specific tasks, and can be used as an external calibration module to access different models, different sampling strategies and different natural language generation tasks. It has good versatility and engineering deployability.

[0085] (6) This invention supports semantic deduplication and effective risk warning, and can output a more compact, interpretable set of predicted answers with risk boundary descriptions to users or downstream systems.

[0086] The technical solution provided in this invention involves: acquiring a target open query and its corresponding calibration dataset; inputting the calibration query into a pre-trained target question-answering model to obtain a set of candidate answers; evaluating the reliability of each candidate answer based on a preset reliability evaluation formula to determine a reliability score for each candidate answer; filtering the candidate answer set based on the candidate calibration threshold and the reliability score for each candidate calibration threshold to obtain a set of predicted answers corresponding to the candidate calibration threshold; determining the empirical loss of coverage for each set of predicted answers and using the minimum candidate calibration threshold for which the empirical loss of coverage satisfies the risk control condition as the target calibration threshold; inputting the target open query into the target question-answering model to obtain a set of optional answers, and filtering the set of optional answers based on the target calibration threshold to determine at least one target answer corresponding to the target open query. The technical solution of this invention addresses the problems of existing large language models where single-point output is susceptible to illusions and the candidate answer set may not contain the correct answer in open query generation. It can first repeatedly sample the calibration query to obtain a candidate answer set, determine the target calibration threshold by performing reliability analysis on the candidate answer set, and then filter the answers corresponding to the target open query based on the target calibration threshold. This can improve the credibility, interpretability and candidate set efficiency of the results in open generation scenarios.

[0087] Figure 7 This is a schematic diagram of a question-answering prediction device based on a large language model provided in an embodiment of the present invention. The embodiment of the present invention can be applied to scenarios of answering open-ended query questions. The device can be implemented by software and / or hardware and integrated into a computer device with application development capabilities.

[0088] like Figure 7 As shown, the question-answering prediction device based on a large language model includes: a data acquisition module 310, a calibration threshold determination module 320, and an answer determination module 330.

[0089] The data acquisition module 310 is used to acquire a target open query and a corresponding calibration dataset; wherein the calibration dataset includes at least one calibration query and a reference answer corresponding to the calibration query; wherein the calibration query and the target open query belong to the same technical field; the calibration threshold determination module 320 is used to input the calibration query into a pre-trained target question-answering model to obtain a corresponding set of candidate answers, and perform reliability analysis on the candidate answers in the set of candidate answers to determine a target calibration threshold; the answer determination module 330 is used to input the target open query into the target question-answering model to obtain a corresponding set of optional answers, and filter the set of optional answers based on the target calibration threshold to determine at least one target answer corresponding to the target open query.

[0090] The technical solution provided by this invention involves obtaining a target open-ended query and a corresponding calibration dataset. The calibration dataset includes at least one calibration query and a corresponding reference answer. The calibration query and the target open-ended query belong to the same technical field. The calibration query is input into a pre-trained target question-answering model to obtain a set of candidate answers. Reliability analysis is performed on the candidate answers in the set to determine a target calibration threshold. The target open-ended query is then input into the target question-answering model to obtain a set of optional answers. Based on the target calibration threshold, the optional answers are filtered to determine at least one target answer corresponding to the target open-ended query. This invention addresses the problems of existing large language models where single-point output is susceptible to illusions and the candidate answer set may not contain correct answers. It first repeatedly samples the calibration query to obtain a set of candidate answers, then performs reliability analysis on the candidate answer set to determine the target calibration threshold, and finally filters the answers corresponding to the target open-ended query based on the target calibration threshold. This improves the credibility, interpretability, and candidate set efficiency of results in open-ended generation scenarios.

[0091] In one optional implementation, the calibration threshold determination module 320 includes a target calibration threshold determination submodule, configured to: evaluate the reliability of each candidate answer based on a preset reliability evaluation formula, and determine the reliability score corresponding to each candidate answer; for each candidate calibration threshold, filter the candidate answer set based on the candidate calibration threshold and the reliability score to obtain a predicted answer set corresponding to the candidate calibration threshold; determine the empirical loss of coverage corresponding to each predicted answer set, and use the minimum candidate calibration threshold for which the empirical loss of coverage satisfies the risk control condition as the target calibration threshold.

[0092] In one optional implementation, the target calibration threshold determination submodule includes: a reliability evaluation unit, configured to: determine a cross-sample consistency parameter based on the semantic similarity between candidate answers in the candidate answer set; determine a self-uncertainty parameter based on the correlation analysis between the candidate answers and the calibration query; perform semantic clustering on the candidate answers in the candidate answer set, and determine the semantic consensus strength based on the data volume comparison results between semantic clusters; and determine the reliability score corresponding to the candidate answer based on the cross-sample consistency parameter, the self-uncertainty parameter, and the semantic consensus strength.

[0093] In one optional implementation, the reliability scoring unit includes a cross-sample consistency parameter determination subunit, configured to: for each current candidate answer, determine the semantic similarity between the current candidate answer and each other candidate answer in the candidate answer set, and use the average of all semantic similarities as the cross-sample consistency parameter corresponding to the current candidate answer.

[0094] In one optional implementation, the reliability scoring unit includes a semantic consensus strength determination subunit, configured to: perform semantic clustering on the candidate answers in the candidate answer set to obtain multiple semantic clusters; for each current candidate answer, compare the data volume of the semantic cluster in which the current candidate answer is located with that of the largest semantic cluster, and use the obtained ratio as the semantic consensus strength of the current candidate answer.

[0095] In one optional implementation, the reliability scoring unit includes a reliability scoring determination subunit, configured to: normalize the cross-sample consistency parameter and the self-uncertainty parameter to obtain a target consistency parameter and a target uncertainty parameter; perform a weighted summation of the target consistency parameter and the target uncertainty parameter to obtain a basic quality score, and use the product of the basic quality score and the semantic consensus strength as the reliability score.

[0096] In an optional implementation, the calibration threshold determination module 320 includes a risk control condition determination module, configured to: substitute each empirical coverage loss into a preset risk control function to obtain a candidate risk control function corresponding to each candidate calibration threshold; and confirm that the candidate calibration threshold satisfies the risk control condition if the risk control function is not greater than the effective risk value, there is a feasible solution, and the lowest achievable risk value is not greater than the user's target risk value.

[0097] In an optional implementation, the question-answering prediction device based on a large language model further includes a risk value adjustment module, configured to: before performing reliability analysis based on the candidate answers in the candidate answer set, perform semantic alignment analysis on the candidate answer set corresponding to each calibration query and the reference answer corresponding to the calibration query, and determine the minimum achievable risk value based on the semantic alignment analysis results; if the minimum achievable risk value is greater than the user's target risk value, adjust the effective risk value so that the adjusted effective risk value is not less than the minimum achievable risk value.

[0098] In an optional implementation, the risk value adjustment module includes a minimum achievable risk value determination submodule, configured to: for each candidate answer set, perform semantic similarity analysis between each candidate answer in the candidate answer set and the reference answer corresponding to the calibration query, and determine the candidate similarity corresponding to each candidate answer; determine the failure indicator corresponding to the candidate answer set based on a preset semantic alignment threshold and the candidate similarity, and use the average of the failure indicators of all candidate answer sets as the minimum achievable risk value.

[0099] The question-answering prediction device based on a large language model provided in this embodiment of the invention can execute the question-answering prediction method based on a large language model provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0100] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 8 A block diagram of an exemplary computer device 12 suitable for implementing embodiments of the present invention is shown. Figure 8 The computer device 12 shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention. The computer device 12 can be any terminal device with computing capabilities and can be configured in a question-answering prediction device based on a large language model.

[0101] like Figure 8 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0102] Bus 18 can be one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0103] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0104] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 8 Not shown; usually referred to as a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0105] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0106] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the computer device 12, and / or with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. Figure 8 As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although... Figure 8 As not shown, it can be used in conjunction with computer device 12 with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0107] Processing unit 16 executes various functional applications and data processing by running programs stored in system memory 28, such as implementing the question-answering prediction method based on a large language model provided in this embodiment of the invention, which includes: Obtain a target open-ended query and its corresponding calibration dataset; wherein the calibration dataset includes at least one calibration query and a reference answer corresponding to the calibration query; wherein the calibration query and the target open-ended query belong to the same technical field; input the calibration query into a pre-trained target question-answering model to obtain a corresponding set of candidate answers, perform reliability analysis on the candidate answers in the set of candidate answers, and determine a target calibration threshold; input the target open-ended query into the target question-answering model to obtain a corresponding set of optional answers, and filter the set of optional answers based on the target calibration threshold to determine at least one target answer corresponding to the target open-ended query.

[0108] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the question-answering prediction method based on a large language model as provided in any embodiment of the present invention, including: Obtain a target open-ended query and its corresponding calibration dataset; wherein the calibration dataset includes at least one calibration query and a reference answer corresponding to the calibration query; wherein the calibration query and the target open-ended query belong to the same technical field; input the calibration query into a pre-trained target question-answering model to obtain a corresponding set of candidate answers, perform reliability analysis on the candidate answers in the set of candidate answers, and determine a target calibration threshold; input the target open-ended query into the target question-answering model to obtain a corresponding set of optional answers, and filter the set of optional answers based on the target calibration threshold to determine at least one target answer corresponding to the target open-ended query.

[0109] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0110] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0111] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0112] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as C, Java, Smalltalk, C++, C#, and Python, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0114] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A question-answering prediction method based on a large language model, characterized in that, include: Obtain a target open-ended query and its corresponding calibration dataset; wherein the calibration dataset includes: at least one calibration query and a reference answer corresponding to the calibration query; wherein the calibration query and the target open-ended query belong to the same technical field; The calibration query is input into the pre-trained target question-answering model to obtain the corresponding candidate answer set. Reliability analysis is performed on the candidate answers in the candidate answer set to determine the target calibration threshold. The target open query is input into the target question-answering model to obtain the corresponding set of optional answers. The set of optional answers is then filtered based on the target calibration threshold to determine at least one target answer corresponding to the target open query.

2. The method according to claim 1, characterized in that, The step of performing reliability analysis based on candidate answers in the candidate answer set to determine the target calibration threshold includes: Based on the preset reliability evaluation formula, the reliability of each candidate answer is evaluated, and the reliability score corresponding to each candidate answer is determined. For each candidate calibration threshold, the candidate answer set is filtered based on the candidate calibration threshold and the reliability score to obtain the predicted answer set corresponding to the candidate calibration threshold; The empirical loss of coverage corresponding to each set of predicted answers is determined, and the minimum candidate calibration threshold for the empirical loss of coverage that satisfies the risk control condition is taken as the target calibration threshold.

3. The method according to claim 2, characterized in that, The reliability evaluation of each candidate answer is performed based on a preset reliability evaluation formula, and a reliability score is determined for each candidate answer, including: Based on the semantic similarity between candidate answers in the candidate answer set, determine the cross-sample consistency parameter; Based on the correlation analysis between the candidate answers and the calibration query, the self-uncertainty parameters are determined; Semantic clustering is performed on the candidate answers in the candidate answer set, and the semantic consensus strength is determined based on the comparison of data volume between semantic clusters; The reliability score corresponding to the candidate answer is determined based on the cross-sample consistency parameter, the self-uncertainty parameter, and the semantic consensus strength.

4. The method according to claim 3, characterized in that, The determination of cross-sample consistency parameters based on the semantic similarity between candidate answers in the candidate answer set includes: For each current candidate answer, the semantic similarity between the current candidate answer and each other candidate answer in the candidate answer set is determined, and the average of all semantic similarities is used as the cross-sample consistency parameter corresponding to the current candidate answer.

5. The method according to claim 3, characterized in that, The step of semantically clustering candidate answers in the candidate answer set and determining the semantic consensus strength based on the data volume comparison results between semantic clusters includes: Semantic clustering is performed on the candidate answers in the candidate answer set to obtain multiple semantic clusters; For each current candidate answer, the data volume of the semantic cluster to which the current candidate answer belongs is compared with that of the largest semantic cluster, and the ratio is used as the semantic consensus strength of the current candidate answer.

6. The method according to claim 3, characterized in that, The process of determining the reliability score corresponding to the candidate answer based on the cross-sample consistency parameter, the self-uncertainty parameter, and the semantic consensus strength includes: The cross-sample consistency parameter and the self-uncertainty parameter are normalized to obtain the target consistency parameter and the target uncertainty parameter; The basic quality score is obtained by weighted summation of the target consistency parameter and the target uncertainty parameter, and the product of the basic quality score and the semantic consensus strength is used as the reliability score.

7. The method according to claim 2, characterized in that, The method for determining the risk control conditions includes: Substitute each empirical loss of coverage into the preset risk control function to obtain the candidate risk control function corresponding to each candidate calibration threshold; If the risk control function has a feasible solution that is not greater than the effective risk value and the lowest achievable risk value is not greater than the user's target risk value, then the candidate calibration threshold does indeed meet the risk control conditions.

8. The method according to claim 1, characterized in that, Before performing reliability analysis based on the candidate answers in the aforementioned candidate answer set, the following steps are also included: Semantic alignment analysis is performed on the candidate answer set corresponding to each calibration query and the reference answer corresponding to the calibration query, and the minimum achievable risk value is determined based on the semantic alignment analysis results; If the minimum achievable risk value is greater than the user's target risk value, the effective risk value is adjusted so that the adjusted effective risk value is not less than the minimum achievable risk value.

9. The method according to claim 8, characterized in that, The step of performing semantic alignment analysis on the candidate answer set corresponding to each calibration query and the reference answer corresponding to the calibration query, and determining the minimum achievable risk value based on the semantic alignment analysis results, includes: For each set of candidate answers, perform semantic similarity analysis between each candidate answer in the set and the reference answer corresponding to the calibration query, and determine the candidate similarity for each candidate answer. The failure indicator value corresponding to the candidate answer set is determined based on the preset semantic alignment threshold and the candidate similarity, and the average value of the failure indicator values ​​of all candidate answer sets is taken as the minimum achievable risk value.

10. A question-answering prediction device based on a large language model, characterized in that, The device includes: A data acquisition module is used to acquire a target open query and a corresponding calibration dataset; wherein, the calibration dataset includes: at least one calibration query and a reference answer corresponding to the calibration query; wherein, the calibration query and the target open query belong to the same technical field; The calibration threshold determination module is used to input the calibration query into the pre-trained target question-answering model to obtain the corresponding candidate answer set, perform reliability analysis on the candidate answers in the candidate answer set, and determine the target calibration threshold. The answer determination module is used to input the target open query into the target question-answering model, obtain the corresponding set of optional answers, and filter the set of optional answers based on the target calibration threshold to determine at least one target answer corresponding to the target open query.

Images