Method, device and electronic equipment for constructing trusted preference answer pairs

Abstract

Embodiments of the present application provide a method and device for constructing a trusted preference answer pair and an electronic device, relating to the technical field of artificial intelligence. The method comprises: using a training sample set to obtain a target preference evaluation model through a response position exchange mechanism; for a candidate recall threshold set, the following operations are respectively performed: based on the output of the target preference evaluation model on the calibration sample set, the number of recalled samples and the number of predicted error samples in the recalled sample set under one candidate recall threshold are counted, and through the inversion of the binomial distribution cumulative distribution function, the upper confidence limit of the error rate corresponding to one candidate recall threshold is determined based on the number of recalled samples, the number of predicted error samples and a preset significance level; the minimum candidate recall threshold corresponding to the upper confidence limit of the error rate smaller than or equal to the preset target error rate is taken as the target recall threshold; and from the candidate answer pair set, a trusted preference answer pair set with a preference prediction probability greater than or equal to the target recall threshold is screened out.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and electronic device for constructing credible preference response pairs. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in scenarios such as dialogue question answering and code generation, how to make the model output conform to human preferences has become one of the core problems in alignment training. The current mainstream approach usually adopts the framework of "humans or models give preference evaluations → train reward models or directly optimize strategies", where the key step is to construct reliable preference-response pairs for multiple candidate answers to the same question.

[0003] In existing technologies, the preference comparison stage typically employs an automated evaluation method based on a general large model as a judge (LLM-as-a-judge). This type of method uses one or more large language models as "judges," taking a user question and two candidate answers as input. The judge model then determines "which one is better" or "whether it's a tie" based on preset evaluation dimensions (such as relevance, correctness, and politeness), and sometimes requires natural language reasoning.

[0004] However, using the above methods, the evaluation results are often highly sensitive to factors such as prompt word design, sampling temperature, and positional bias. The same question may elicit inconsistent conclusions under different prompts or multiple calls, and is easily influenced by biases such as the position of the answer and the model family itself, leading to a systematic deviation from real human preferences. Furthermore, existing technologies generally use methods such as voting and heuristic confidence thresholds to roughly filter uncertain samples, only making a coarse-grained trade-off between "retaining as many samples as possible" and "discarding some low-confidence samples through simple thresholds." They cannot explicitly set the target preference accuracy (e.g., preference accuracy greater than 95%) during the design phase, lack a unified statistical testing framework, cannot provide quantitative guarantees on the error rate of preference labels at a given significance level, and are difficult to pre-set preference accuracy according to task requirements.

[0005] Therefore, existing technologies still fall short of meeting the needs when constructing a set of preference-response pairs that are highly reliable, have quantifiable credibility, and can flexibly adjust the "precision-recall" tradeoff according to the application scenario. Summary of the Invention

[0006] This application provides a method, apparatus, and electronic device for constructing credible preference response pairs, which are used to construct a set of preference response pairs with high reliability, quantifiable credibility, and the ability to flexibly adjust the "precision-recall" tradeoff according to the application scenario. This allows the preference determination process to set the target preference accuracy as needed, providing high-quality and controllable reliability preference data for the alignment training of large language models.

[0007] In a first aspect, embodiments of this application provide a method for constructing credible preference response pairs, the method comprising: Obtain the training sample set and calibration sample set; Using a training sample set, the preference evaluation model to be trained is trained through a response position swapping mechanism to obtain the target preference evaluation model; For each candidate recall threshold in the candidate recall threshold set, perform the following operations: Based on the output of the target preference evaluation model to the calibration sample set, count the number of recalled samples and the number of mispredicted samples in the recalled sample set under a candidate recall threshold, and determine the upper confidence bound of the error rate corresponding to a candidate recall threshold based on the number of recalled samples, the number of mispredicted samples and the preset significance level by inverting the binomial cumulative distribution function. The smallest candidate recall threshold in the candidate recall threshold set whose error rate upper confidence boundary is less than or equal to the preset target error rate is taken as the target recall threshold. From the candidate answer pair set, a set of credible preference answer pairs with preference prediction probabilities greater than or equal to the target recall threshold is selected. The preference prediction probability is obtained by inputting the corresponding candidate answer pairs into the target preference evaluation model.

[0008] In an optional embodiment, each training sample in the training sample set contains a sample-response pair and a first true preference label, the first true preference label being used to indicate the preferred sample response among the two sample responses contained in the sample-response pair. Each calibration sample in the calibration sample set contains a sample-response pair and a second true preference label, the second true preference label being used to indicate the preferred sample response among the two sample responses contained in the sample-response pair; The first true preference label is obtained based on a preference rating label, and the second true preference label is obtained based on at least two preference rating labels.

[0009] In one optional embodiment, obtaining the training sample set and the calibration sample set includes: For each sample question, perform the following operations: Use multiple dialogue models to generate multiple sample answers for a sample question, and combine the multiple sample answers in pairs to form multiple sample answer pairs corresponding to a sample question. Each sample answer pair includes: a sample question, a first sample answer, and a second sample answer. The first sample answer and the second sample answer are different. Each sample answer pair is labeled with a preference score to obtain a labeled dataset, and the standard dataset is divided into a training sample set and a calibration sample set.

[0010] In one optional embodiment, a training sample set is used to train the preference evaluation model to be trained through a response position swapping mechanism to obtain the target preference evaluation model, including: The preference evaluation model to be trained is iteratively trained based on the training sample set to obtain the target preference evaluation model. During one iteration of training, the following operations are performed: Construct a sub-training sample group of the selected target training sample; wherein, the sub-training sample group includes: a first sub-training sample and a second sub-training sample, and the two samples in the target training sample have different positional relationships in the first sub-training sample and the second sub-training sample; The first sub-training sample and the second sub-training sample are respectively input into the preference evaluation model to be trained to obtain the first sub-preference prediction probability corresponding to the first sub-training sample and the second sub-preference prediction probability corresponding to the second training sample. Based on the first sub-preference prediction probability and the second sub-preference prediction probability, the basic cross-entropy loss and mean squared error loss are obtained, and the comprehensive loss is determined based on the basic cross-entropy loss and the mean squared error loss. The first sub-preference prediction probability is the preference probability of the sample answer at the first input position of the first sub-training sample, and the second sub-preference prediction probability is the preference probability of the sample answer at the second input position of the second sub-training sample. Based on the comprehensive loss, the model parameters of the preference evaluation model to be trained are adjusted.

[0011] In one optional embodiment, the sample answer at the first input position of the first sub-training sample is a preferred sample answer, and the sample answer at the second input position of the first sub-training sample is a non-preferred sample answer. The sample answer at the first input position of the second sub-training sample is a non-preference sample answer, and the sample answer at the second input position of the second sub-training sample is a preference sample answer; The first input position is before the second input position, and the preferred sample answer is the preferred sample answer among the two sample answers indicated by the first true preference label in the target training sample.

[0012] In an optional embodiment, based on the output of the target preference evaluation model to the calibration sample set, the number of recalled samples at a candidate recall threshold and the number of incorrectly predicted samples in the recalled sample set are counted, including: Each calibration sample in the calibration sample set is input into the target preference evaluation model to obtain the preference prediction result and preference prediction probability for each calibration sample in the calibration sample set. From the calibration sample set, select each calibration sample whose preference prediction probability is greater than or equal to a candidate recall threshold, and use it as the recalled sample set, and count the number of recalled samples in the recalled sample set. If the number of recalled samples is greater than or equal to the preset sample coverage threshold, the number of incorrectly predicted samples in the recalled sample set is determined based on the preference prediction results and true preference labels of each calibration sample in the recalled sample set.

[0013] In one optional embodiment, each calibration sample in the calibration sample set is input into the target preference evaluation model to obtain the preference prediction result and preference prediction probability for each calibration sample in the calibration sample set, including: For each calibration sample in the calibration sample set, perform the following operations: Construct a sub-calibration sample group for a calibration sample; wherein the sub-calibration sample group includes: a first sub-calibration sample and a second sub-calibration sample, and the two samples in a calibration sample have different positional relationships in the first and second sub-calibration samples; The first and second sub-calibration samples are input into the target preference evaluation model to obtain the third and fourth sub-preference prediction probabilities corresponding to the first sub-calibration sample, and the fifth and sixth sub-preference prediction probabilities corresponding to the second sub-calibration sample. The third sub-preference prediction probability is the preference probability of the sample answer at the first input position of the first sub-calibration sample, the fourth sub-preference prediction probability is the preference probability of the sample answer at the second input position of the first sub-training sample, the fifth sub-preference prediction probability is the preference probability of the sample answer at the first input position of the second sub-calibration sample, and the sixth sub-preference prediction probability is the preference probability of the sample answer at the second input position of the second sub-training sample. The first preference prediction probability is obtained by averaging the third and sixth sub-preference prediction probabilities in a calibration sample, and the second preference prediction probability is obtained by averaging the fourth and fifth sub-preference prediction probabilities in a calibration sample, and the second preference prediction probability is obtained by averaging the second and fifth sub-preference prediction probabilities in a calibration sample. The first preference prediction probability corresponding to the first sample answer and the second preference prediction probability corresponding to the second sample answer are compared to obtain the preference prediction result and preference prediction probability of a calibration sample.

[0014] In one optional embodiment, an upper confidence bound for the error rate corresponding to a candidate recall threshold is determined based on the number of recalled samples, the number of mispredicted samples, and a preset significance level, using the inversion method of the binomial cumulative distribution function. This includes: Based on the number of incorrectly predicted samples and the number of recalled samples, determine the empirical error rate corresponding to a candidate recall threshold; By inverting the cumulative distribution function of the binomial distribution, an upper confidence bound for the error rate corresponding to a candidate recall threshold is determined based on the empirical error rate and the preset significance level. The preset significance level is a preset statistical risk control parameter.

[0015] Secondly, embodiments of this application also provide an apparatus for constructing credible preference answer pairs, the apparatus comprising: The acquisition module is used to acquire the training sample set and the calibration sample set; The training module is used to train the preference evaluation model to be trained using the training sample set and the response position exchange mechanism to obtain the target preference evaluation model. The processing module is used to perform the following operations for each candidate recall threshold in the candidate recall threshold set: based on the output of the target preference evaluation model to the calibration sample set, count the number of recalled samples and the number of predicted incorrect samples in the recalled sample set under a candidate recall threshold, and determine the upper confidence bound of the error rate corresponding to a candidate recall threshold based on the number of recalled samples, the number of predicted incorrect samples and the preset significance level by inverting the binomial cumulative distribution function; The determination module is used to select the smallest candidate recall threshold in the candidate recall threshold set whose corresponding error rate upper confidence boundary is less than or equal to the preset target error rate, as the target recall threshold. The filtering module is used to filter out a set of credible preference answer pairs from the candidate answer pair set whose preference prediction probability is greater than or equal to the target recall threshold. The preference prediction probability is obtained by inputting the corresponding candidate answer pair into the target preference evaluation model.

[0016] In an optional embodiment, each training sample in the training sample set contains a sample-response pair and a first true preference label, the first true preference label being used to indicate the preferred sample response among the two sample responses contained in the sample-response pair. Each calibration sample in the calibration sample set contains a sample-response pair and a second true preference label, the second true preference label being used to indicate the preferred sample response among the two sample responses contained in the sample-response pair; The first true preference label is obtained based on a preference rating label, and the second true preference label is obtained based on at least two preference rating labels.

[0017] In an optional embodiment, when acquiring the training sample set and the calibration sample set, the acquisition module is further configured to: For each sample question, perform the following operations: Use multiple dialogue models to generate multiple sample answers for a sample question, and combine the multiple sample answers in pairs to form multiple sample answer pairs corresponding to a sample question. Each sample answer pair includes: a sample question, a first sample answer, and a second sample answer. The first sample answer and the second sample answer are different. Each sample answer pair is labeled with a preference score to obtain a labeled dataset, and the standard dataset is divided into a training sample set and a calibration sample set.

[0018] In an optional embodiment, when training a training sample set and training the preference evaluation model to be trained through a response position swapping mechanism to obtain the target preference evaluation model, the training module is further used for: The preference evaluation model to be trained is iteratively trained based on the training sample set to obtain the target preference evaluation model. During one iteration of training, the following operations are performed: Construct a sub-training sample group of the selected target training sample; wherein, the sub-training sample group includes: a first sub-training sample and a second sub-training sample, and the two samples in the target training sample have different positional relationships in the first sub-training sample and the second sub-training sample; The first sub-training sample and the second sub-training sample are respectively input into the preference evaluation model to be trained to obtain the first sub-preference prediction probability corresponding to the first sub-training sample and the second sub-preference prediction probability corresponding to the second training sample. Based on the first sub-preference prediction probability and the second sub-preference prediction probability, the basic cross-entropy loss and mean squared error loss are obtained, and the comprehensive loss is determined based on the basic cross-entropy loss and the mean squared error loss. The first sub-preference prediction probability is the preference probability of the sample answer at the first input position of the first sub-training sample, and the second sub-preference prediction probability is the preference probability of the sample answer at the second input position of the second sub-training sample. Based on the comprehensive loss, the model parameters of the preference evaluation model to be trained are adjusted.

[0019] In one optional embodiment, the sample answer at the first input position of the first sub-training sample is a preferred sample answer, and the sample answer at the second input position of the first sub-training sample is a non-preferred sample answer. The sample answer at the first input position of the second sub-training sample is a non-preference sample answer, and the sample answer at the second input position of the second sub-training sample is a preference sample answer; The first input position is before the second input position, and the preferred sample answer is the preferred sample answer among the two sample answers indicated by the first true preference label in the target training sample.

[0020] In an optional embodiment, when calculating the number of recalled samples and the number of incorrectly predicted samples in the recalled sample set based on the output of the target preference evaluation model to the calibration sample set, the processing module is further configured to: Each calibration sample in the calibration sample set is input into the target preference evaluation model to obtain the preference prediction result and preference prediction probability for each calibration sample in the calibration sample set. From the calibration sample set, select each calibration sample whose preference prediction probability is greater than or equal to a candidate recall threshold, and use it as the recalled sample set, and count the number of recalled samples in the recalled sample set. If the number of recalled samples is greater than or equal to the preset sample coverage threshold, the number of incorrectly predicted samples in the recalled sample set is determined based on the preference prediction results and true preference labels of each calibration sample in the recalled sample set.

[0021] In an optional embodiment, when inputting each calibration sample in the calibration sample set into the target preference evaluation model to obtain the preference prediction result and preference prediction probability of each calibration sample in the calibration sample set, the processing module is further configured to: For each calibration sample in the calibration sample set, perform the following operations: Construct a sub-calibration sample group for a calibration sample; wherein the sub-calibration sample group includes: a first sub-calibration sample and a second sub-calibration sample, and the two samples in a calibration sample have different positional relationships in the first and second sub-calibration samples; The first and second sub-calibration samples are input into the target preference evaluation model to obtain the third and fourth sub-preference prediction probabilities corresponding to the first sub-calibration sample, and the fifth and sixth sub-preference prediction probabilities corresponding to the second sub-calibration sample. The third sub-preference prediction probability is the preference probability of the sample answer at the first input position of the first sub-calibration sample, the fourth sub-preference prediction probability is the preference probability of the sample answer at the second input position of the first sub-training sample, the fifth sub-preference prediction probability is the preference probability of the sample answer at the first input position of the second sub-calibration sample, and the sixth sub-preference prediction probability is the preference probability of the sample answer at the second input position of the second sub-training sample. The first preference prediction probability is obtained by averaging the third and sixth sub-preference prediction probabilities in a calibration sample, and the second preference prediction probability is obtained by averaging the fourth and fifth sub-preference prediction probabilities in a calibration sample, and the second preference prediction probability is obtained by averaging the second and fifth sub-preference prediction probabilities in a calibration sample. The first preference prediction probability corresponding to the first sample answer and the second preference prediction probability corresponding to the second sample answer are compared to obtain the preference prediction result and preference prediction probability of a calibration sample.

[0022] In an optional embodiment, when determining the upper confidence bound of the error rate corresponding to a candidate recall threshold based on the number of recalled samples, the number of mispredicted samples, and a preset significance level by inverting the cumulative distribution function of the binomial distribution, the processing module is further configured to: Based on the number of incorrectly predicted samples and the number of recalled samples, determine the empirical error rate corresponding to a candidate recall threshold; By inverting the cumulative distribution function of the binomial distribution, an upper confidence bound for the error rate corresponding to a candidate recall threshold is determined based on the empirical error rate and the preset significance level. The preset significance level is a preset statistical risk control parameter.

[0023] Thirdly, embodiments of this application also provide an electronic device, including: Processor; and Stored program memory, The program includes instructions that, when executed by a processor, cause the processor to perform the method for constructing trustworthy preference answer pairs as described in the first aspect.

[0024] Fourthly, embodiments of this application also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method for constructing trustworthy preference answer pairs as described in the first aspect.

[0025] Fifthly, this application provides a computer program product that, when invoked by a computer, causes the computer to execute the steps of the method for constructing trustworthy preference answer pairs as described in the first aspect.

[0026] The beneficial effects of this application are as follows: In the method for constructing credible preference answer pairs provided in this application embodiment, a training sample set and a calibration sample set are obtained. Then, using the training sample set, the preference evaluation model to be trained is trained through an answer position swapping mechanism to obtain the target preference evaluation model. Then, for each candidate recall threshold in the candidate recall threshold set, the following operations are performed: based on the output of the target preference evaluation model to the calibration sample set, the number of recalled samples and the number of predicted incorrect samples in the recalled sample set are counted under a candidate recall threshold. And through the inversion of the binomial cumulative distribution function, based on the number of recalled samples, the number of predicted incorrect samples, and the preset significance level, the upper confidence bound of the error rate corresponding to a candidate recall threshold is determined. Finally, the smallest candidate recall threshold in the candidate recall threshold set whose corresponding upper confidence bound of the error rate is less than or equal to the preset target error rate is taken as the target recall threshold. Thus, from the candidate answer pair set, a set of credible preference answer pairs with preference prediction probabilities greater than or equal to the target recall threshold is selected. The preference prediction probability is obtained by inputting the corresponding candidate answer pair into the target preference evaluation model. In this way, on the one hand, the adaptive recall threshold solution based on the calibration sample set and statistical tests automatically obtains the target recall threshold under the target error rate and significance level that can be set by the user. It provides a controllable constraint on the error rate of the recalled preference labels, avoiding the instability and uninterpretability brought about by traditional LLM-as-a-judge or heuristic threshold screening. From the application level, it provides a quantifiable and provable upper limit guarantee of the error rate for the finally selected set of credible preference pairs (for example, guaranteeing that the error rate is not higher than 5% with high confidence). This transforms the entire construction process from a "black box" operation that depends on empirical thresholds into a reliable technical solution with clear objectives, controllable process, and verifiable results. This provides a high-quality and highly reliable data foundation for downstream reward model training or large language model alignment. On the other hand, by training the preference evaluation model to be trained through the response position swapping mechanism, the judgment bias caused by the input order is actively weakened at the model level. This solves the order bias problem caused by position encoding, which makes the "first-appearing response more likely to be preferred". This ensures that the model maintains consistent confidence in the same true better response under both arrangements, thereby weakening the dependence on the response input order and significantly improving the consistency and robustness of preference judgment.

[0027] Furthermore, other features and advantages of this application will be set forth in the following description and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described herein are used to provide a further understanding of this application, constitute a part of this application, and do not constitute an improper limitation of this application. In the accompanying drawings: Figure 1 This is a schematic diagram of an optional system architecture applicable to the embodiments of this application.

[0029] Figure 2 This is a schematic diagram illustrating the implementation process of a method for constructing trustworthy preference answer pairs provided in an embodiment of this application.

[0030] Figure 3 This is a logical diagram illustrating the determination of a target recall threshold, as provided in an embodiment of this application.

[0031] Figure 4 This is a schematic diagram of a device for constructing a trustworthy preference response pair, provided in an embodiment of this application.

[0032] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0034] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0035] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0036] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0037] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0038] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0039] (1) Large Language Model (LLM): A large-scale deep learning model trained on massive text data. Its core capability is to understand the semantics, logic and patterns of human language, and at the same time, it can generate text content that conforms to language rules.

[0040] (2) General large model as a judge (LLM-as-a-judge): refers to the technical approach of using one or more large language models to evaluate / compare the quality of "question + candidate answer" and output preference conclusions.

[0041] (3) Supervised Fine-Tuning (SFT): refers to supervising the training of a pre-trained large model with labeled data (usually with the goal of minimizing cross-entropy) so that it learns to output specific tasks.

[0042] (4) Mean Squared Error (MSE): A common loss for regression / consistency constraints; this application uses it to constrain the confidence levels to be close under two permutations (symmetry constraint).

[0043] Based on the above explanations of terms and related terminology, the design concept of the embodiments of this application will be briefly introduced below: With the widespread application of LLM in scenarios such as dialogue question answering and code generation, how to make the model output conform to human preferences has become one of the core problems in alignment training. The current mainstream approach usually adopts the framework of "human or model gives preference evaluation → train reward model or directly optimize strategy", where the key step is to construct reliable preference answer pairs for multiple candidate answers to the same question.

[0044] In existing technologies, the following types of solutions are typically used in the preference comparison stage: (1) Automatic evaluation method based on LLM-as-a-judge: This type of method uses one or more large language models as "judges". The user inputs a question and two candidate answers, and the judge model gives a judgment of "which one is better" or "whether it is a tie" according to preset evaluation dimensions (such as relevance, correctness, politeness, etc.). Sometimes, natural language reasons are also required. Depending on the output format, it can be divided into pointwise evaluation (scoring each answer individually), pairwise evaluation (comparing two answers at the same time), and listwise evaluation (comparing multiple answers at once). This type of method does not require manual scoring of each answer and has been widely used to construct synthetic preference data and online automatic evaluation.

[0045] (2) Preference comparison method based on discriminative reward model: Discriminative reward models typically add a numerical regression head to a pre-trained language model, mapping the "question-answer" pair to a real-valued score to measure how well the answer matches human preferences. During the training phase, pairwise comparison loss methods such as Bradley-Terry are often used to convert the "good answer-bad answer" label pairs into cross-entropy loss based on score differences, enabling the reward model to output higher scores for the winning answer given a question. During the inference phase, when comparing two candidate answers, they are generally input into the reward model to obtain two scores, and then the superiority or inferiority is determined by comparing the scores. This type of method has become a standard component in traditional RLHF pipelines.

[0046] (3) Preference comparison method based on pairwise generative reward model: In order to better utilize the generative capabilities of large models, in recent years, schemes have emerged that design reward models as generative models. After inputting "question + two candidate answers", such models no longer output only a numerical score, but generate natural language containing preference conclusions and reasoning processes. For example, they first generate an analysis comparing the two answers, and then give the final judgment of "preferring answer A / B or difficult to distinguish"; or after generating hidden reasoning trajectories, they derive preference labels. Generative reward models can model preference labels and explanatory content at the same time, and are also convenient for self-supervised or semi-supervised training using unlabeled data. Therefore, they have received increasing attention in the field of alignment.

[0047] Although the above techniques have achieved some results in practical applications, there are still significant shortcomings in constructing credible preference response pairs, mainly reflected in: (1) Insufficient stability and controllability of LLM-as-a-judge evaluation results: When LLM acts as a judge, the evaluation results are often sensitive to factors such as prompt word design, sampling temperature, and positional bias. The same question-answer pair may yield inconsistent conclusions under different prompts or multiple calls, and is easily affected by biases such as answer position and its own model family, resulting in a systematic deviation from real human preferences. Existing works generally use voting, heuristic confidence thresholds, etc. to roughly filter uncertain samples, but lack a unified statistical testing framework, and cannot make a quantitative guarantee on the "error rate of preference labels" at a given significance level, making it difficult to preset the preference accuracy rate according to task requirements.

[0048] (2) Scale drift and noise sensitivity of discriminative reward models: Discriminative reward models learn relative preferences through pairwise comparison loss during training, but in the inference stage, they are usually used in the form of "single score + score comparison", which will lead to scale inconsistency and calibration difficulties across problems and scenarios: the true preference strength represented by the same score difference is not consistent on different problems, and the numerical output of the reward model is also prone to drift during iterative optimization.

[0049] (3) Pairwise generative reward models lack explicit statistical confidence modeling: Although generative reward models can provide natural language reasoning processes and preference conclusions, their "preference A or preference B" output is usually a single-point decision obtained through one or a small number of samples, and the implicit probability distribution behind it is difficult to directly observe and calibrate. Existing works mostly use repeated sampling voting or heuristic screening based on the confidence phrases reported by the model, but these methods cannot guarantee that the true error rate of preference labels is strictly constrained under the set confidence threshold.

[0050] Therefore, whether it's LLM-as-a-judge, discriminative, or generative reward models, they can typically only make a coarse-grained trade-off between "retaining as many samples as possible" and "discarding some low-confidence samples through a simple threshold." They cannot explicitly set a target preference accuracy (e.g., 95%) during the design phase. Preference accuracy refers to the proportion of correctly labeled samples in the selected preference-response pair set. Existing technologies also lack a system solution that incorporates self-calibration of the inference process, and lack a secondary review and contradiction detection mechanism for the natural language inference chain provided by the model. Therefore, when it comes to constructing a set of preference-response pairs that requires high reliability, quantifiable credibility, and the ability to flexibly adjust the "precision-recall" trade-off according to the application scenario, existing technologies still fall short. There is an urgent need for a method to construct credible preference pairs that, based on pairwise generative reward models, incorporates statistical testing and multi-stage inference calibration, and can preset preference accuracy.

[0051] In view of this, this application provides a method for constructing credible preference response pairs, which enables the preference determination process to set the target preference accuracy as needed, providing high-quality, controllable, and reliable preference data for the alignment training of large language models. Specifically, this may include: first, acquiring a training sample set and a calibration sample set; then, using the training sample set, training the preference evaluation model to be trained through an answer position swapping mechanism to obtain the target preference evaluation model; next, for each candidate recall threshold in the candidate recall threshold set, performing the following operations: based on the output of the target preference evaluation model to the calibration sample set, counting the number of recalled samples and the number of incorrectly predicted samples in the recalled sample set under a candidate recall threshold, and determining the upper confidence bound of the error rate corresponding to a candidate recall threshold based on the number of recalled samples, the number of incorrectly predicted samples, and a preset significance level through the inversion of the binomial cumulative distribution function; finally, taking the smallest candidate recall threshold in the candidate recall threshold set whose corresponding upper confidence bound of the error rate is less than or equal to the preset target error rate as the target recall threshold; thereby selecting a set of credible preference answer pairs from the candidate answer pair set whose preference prediction probability is greater than or equal to the target recall threshold, where the preference prediction probability is obtained by inputting the corresponding candidate answer pair into the target preference evaluation model.

[0052] By adopting the above approach, on the one hand, adaptive recall threshold solution based on calibration sample set and statistical test automatically obtains target recall threshold under user-configurable target error rate and significance level, and provides controllable constraint on the error rate of recalled preference labels. This avoids the instability and uninterpretability caused by traditional LLM-as-a-judge or heuristic threshold screening. From the application level, it provides a quantifiable and provable upper limit guarantee of error rate for the final selected set of credible preference pairs (e.g., guaranteeing the error rate is no higher than 5% with high confidence). This transforms the entire construction process from a "black box" operation that relies on empirical thresholds into a reliable technical solution with clear objectives, controllable process, and verifiable results. This provides a high-quality and highly reliable data foundation for downstream reward model training or large language model alignment. On the other hand, by training the preference evaluation model to be trained through a response position swapping mechanism, the model actively weakens the judgment bias caused by the input order, solving the order bias problem of "first-appearing responses are more likely to be preferred" caused by positional encoding. This ensures that the model maintains consistent confidence in the same truly better response under both permutations, thereby reducing the dependence on the input order of responses and significantly improving the consistency and robustness of preference judgments. This addresses the two core problems of insufficient stability and uncontrollable reliability in preference judgments in existing technologies.

[0053] In particular, the preferred embodiments of this application will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

[0054] See Figure 1 The diagram illustrates an optional system architecture applicable to an embodiment of this application. This system architecture may include: terminal devices (101a, 101b) and server 102. The terminal devices (101a, 101b) and server 102 can interact via a communication network. The communication network can employ wireless communication and wired communication methods. For example, the terminal devices (101a, 101b) can access the network and communicate with server 102 via cellular mobile communication technology. This cellular mobile communication technology may include, for example, 5th generation mobile networks (5G) technology or next-generation mobile communication technology. Optionally, the terminal devices (101a, 101b) can access the network and communicate with server 102 via short-range wireless communication. This short-range wireless communication method may include, for example, wireless fidelity (Wi-Fi) technology.

[0055] This application embodiment does not impose any limitation on the number of communication devices involved in the above system architecture. For example, the above system architecture may include more terminal devices, or it may include fewer terminal devices, or it may also include other network devices. Figure 1 As shown, only terminal devices (101a, 101b) and server 102 are described as examples. The following is a brief introduction to the above communication devices and their respective functions.

[0056] A terminal device (101a, 101b) is a device that can provide voice and / or data connectivity to a user, and may be a device that supports wired and / or wireless connections.

[0057] For example, terminal devices (101a, 101b) may include, but are not limited to: mobile phones, tablets, laptops, handheld computers, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminal devices in industrial control, wireless terminal devices in autonomous driving, wireless terminal devices in smart grids, wireless terminal devices in transportation safety, wireless terminal devices in smart cities, or wireless terminal devices in smart homes, etc.

[0058] In addition, the terminal devices (101a, 101b) may have related clients installed. These clients may be software, such as applications (APPs), browsers, short video software, or web pages, mini-programs, etc. It should be noted that the terminal devices (101a, 101b) in this application embodiment may be the clients related to the construction of trustworthy preference answer pairs mentioned above.

[0059] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0060] Optionally, a pre-trained target preference evaluation model can be deployed on server 102. After obtaining the candidate answer pair set, server 102 can input the candidate answer pair set into the target preference evaluation model to generate the preference prediction probability of the candidate answer pair set, so as to select a set of credible preference answer pairs with preference prediction probabilities greater than or equal to the target recall threshold from the candidate answer pair set.

[0061] The following describes the method for constructing trustworthy preference answer pairs provided by exemplary embodiments of this application, in conjunction with the above-described system architecture and with reference to the accompanying drawings. It should be noted that the above-described system architecture is only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0062] See Figure 2 The diagram illustrates the implementation flow of a method for constructing trustworthy preference answer pairs according to an embodiment of this application. Taking a server as an example, the specific implementation flow of this method is as follows: S20: Obtain the training sample set and calibration sample set.

[0063] The calibration sample set is of higher quality than the training sample set. The training sample set is used to supervise the fine-tuning of the pairwise generative preference evaluation model, while the calibration sample set is used for subsequent statistical tests and recall threshold solving.

[0064] In this embodiment, each training sample in the training sample set includes a sample answer pair and a first true preference label. The first true preference label indicates the preferred sample answer among the two sample answers contained in the sample answer pair. Each calibration sample in the calibration sample set includes a sample answer pair and a second true preference label. The second true preference label indicates the preferred sample answer among the two sample answers contained in the sample answer pair. The first true preference label is obtained based on a preference score label, and the second true preference label is obtained based on at least two preference score labels. The sample answer pair includes: a sample question and two different sample answers (first sample answer and second sample answer), denoted as (q, a, b), where q is the sample question, a is the first sample answer, and b is the second sample answer.

[0065] Optionally, in this embodiment of the application, a possible implementation is provided for obtaining the training sample set and the calibration sample set, specifically by performing the following operations: S200: For each sample question, perform the following operations: use multiple dialogue models to generate multiple sample answers for a sample question, and combine the multiple sample answers in pairs to form multiple sample answer pairs corresponding to a sample question.

[0066] A sample response pair consists of a sample question, a first sample response, and a second sample response, where the first and second sample responses are different. Multiple dialogue models can be dialogue models of different sizes, architectures, or training strategies. The dialogue models are large language models, and the sample questions are natural language questions.

[0067] For example, for a sample question q1, multiple dialogue models each generate 5 sample answers for q1. The 5 sample answers include: m1, m2, m3, m4, and m5. The 5 sample answers are combined in pairs to obtain 10 sample answer pairs corresponding to q1. The multiple sample answer pairs are: (q1, m1, m2), (q1, m1, m3), (q1, m1, m4), (q1, m1, m5), (q1, m2, m3), (q1, m2, m4), (q1, m2, m5), (q1, m3, m4), (q1, m3, m5), (q1, m4, m5).

[0068] S201: Perform preference scoring and labeling on each sample answer pair to obtain a labeled dataset, and divide the standard dataset into a training sample set and a calibration sample set.

[0069] In this embodiment, a portion of the obtained sample answer pairs is extracted, and human annotators are organized to perform preference scoring and annotation. For each sample answer pair, the annotators comprehensively judge based on dimensions such as usefulness and harmlessness to determine which sample answer they prefer, thus obtaining the true preference label for each sample answer pair (whether the first sample answer is better or the second sample answer is better), resulting in an annotated dataset. Then, the annotation data in the annotated dataset that were annotated by a single person is used as the training sample set, and the annotation data in the annotated dataset that were annotated by multiple people is used as the calibration sample set. In other words, during preference scoring and annotation, a portion of the sample answer pairs are annotated by a single person to obtain the training sample set, while a portion of the sample answer pairs are cross-annotated by multiple people (i.e., multiple preference scores are obtained for a sample answer pair, and the sample answer with the most preference scores is considered the better sample answer), resulting in the calibration sample set.

[0070] In this way, using single-person annotation to generate a large training set aims to cover a wide sample space at a reasonable cost, ensuring that the model can learn rich preference patterns. Meanwhile, using more rigorous multi-person cross-annotation to generate a small but high-quality calibration set ensures that the gold standard data used for statistical testing has extremely high reliability. This makes the upper confidence bound of the error rate calculated based on this calibration set and the final target recall threshold more reliable and accurate, fundamentally enhancing the rigor of the entire statistical assurance process.

[0071] S21: Using the training sample set, the preference evaluation model to be trained is trained through the response position exchange mechanism to obtain the target preference evaluation model.

[0072] The preference evaluation model to be trained can be a large language model. The output of the preference evaluation model uses a dedicated preference token, where [[A]] indicates a preference for the first answer; [[B]] indicates a preference for the second answer. This embodiment does not design a tie token; for samples that are difficult to distinguish, a recall threshold will be used during the statistical testing phase to achieve a "non-recall" effect.

[0073] In this embodiment, the preset significance level is a preset statistical risk control parameter that defines the risk of inference failure that is willing to be borne when using a limited calibration sample to infer the overall data quality.

[0074] Optionally, in this embodiment of the application, when training the preference evaluation model to be trained using a training sample set and a response position swapping mechanism to obtain the target preference evaluation model, iterative training is performed on the preference evaluation model to be trained based on the training sample set to obtain the target preference evaluation model. In one iterative training process, the following operations are performed: SA1: Construct a subgroup of training samples from the selected target training samples.

[0075] The sub-training sample group includes a first sub-training sample and a second sub-training sample. The two samples in the target training sample have different positional relationships in the first and second sub-training samples. The target training sample is any training sample in the training sample set.

[0076] In this embodiment, the sample answer at the first input position of the first sub-training sample is the preferred sample answer, and the sample answer at the second input position of the first sub-training sample is the non-preferred sample answer; the sample answer at the first input position of the second sub-training sample is the non-preferred sample answer, and the sample answer at the second input position of the second sub-training sample is the preferred sample answer; the first input position is before the second input position, and the preferred sample answer is the preferred sample answer among the two sample answers indicated by the first true preference label in the target training sample.

[0077] For example, suppose the target training sample is (q, good, bad), where good refers to the preferred answer among two sample answers in the target training sample, and bad refers to the non-preferred answer among two sample answers in the target training sample; then the first sub-training sample of the target training sample is (q, a good b bad The position of the preference sample's answer is at the first input position; the second sub-training sample of the target training sample is (q, a). bad (bgood), the position of the preferred sample's answer is in the second input position; the two sub-training samples are semantically equivalent, only the order of the two sample answers is reversed.

[0078] SA2: Input the first sub-training sample and the second sub-training sample into the preference evaluation model to be trained, respectively, to obtain the first sub-preference prediction probability corresponding to the first sub-training sample and the second sub-preference prediction probability corresponding to the second training sample.

[0079] Wherein, the first sub-preference prediction probability is the preference probability of the sample answer at the first input position of the first sub-training sample, and the second sub-preference prediction probability is the preference probability of the sample answer at the second input position of the second sub-training sample.

[0080] For example, the preference probability output by the preference evaluation model for the sample answer at the first input position of the first sub-training sample is denoted as p1, and the preference probability output by the preference evaluation model for the sample answer at the second input position of the second sub-training sample is denoted as p2.

[0081] In this way, by constructing two sub-training samples (good-bad, bad-good) with opposite input order for the target training samples and forcing the model to output high confidence for the "good answer" in both orders, it is possible to effectively counteract the problem that may always tend to choose the first answer due to the fixed positional encoding in the Transformer architecture.

[0082] SA3: Based on the prediction probabilities of the first and second sub-preferences, the basic cross-entropy loss and mean squared error loss are obtained, and the comprehensive loss is determined based on the basic cross-entropy loss and mean squared error loss.

[0083] In this embodiment, the basic cross-entropy loss and mean squared error loss are obtained based on the logarithm of the first sub-preference prediction probability and the logarithm of the second sub-preference prediction probability. Then, the comprehensive loss is calculated based on the basic cross-entropy loss, mean squared error loss, and hyperparameters.

[0084] The formula for calculating the basic cross-entropy loss can be expressed as: L ce =-logp1-logp2; The formula for calculating the mean squared error loss can be expressed as: L pos = (logp1 - logp2) 2 Where logp1 is the prediction probability of the first sub-preference and logp2 is the prediction probability of the second sub-preference. The combined loss can be expressed as: L = L ce +βL pos , where β is a hyperparameter.

[0085] SA4: Adjust the model parameters of the preference evaluation model to be trained based on the comprehensive loss.

[0086] In this embodiment of the application, after obtaining the comprehensive loss, the model parameters of the preference evaluation model to be trained are adjusted based on the comprehensive loss.

[0087] In this way, the basic cross-entropy loss ensures that the model can give the correct preference category under both permutations of the first and second sub-training samples, while the MSE loss forces the model to keep the confidence of the same "true better answer" close under both permutations, reducing the dependence on specific positions. Combining the basic cross-entropy loss and the mean squared error symmetric loss, the model is directly constrained from the optimization objective, so that the preference judgment logic it learns is independent of the input order, thereby significantly improving the consistency and stability of the model's judgment, which is a prerequisite for obtaining reliable preference prediction.

[0088] S22: For each candidate recall threshold in the candidate recall threshold set, perform the following operations: Based on the output of the target preference evaluation model to the calibration sample set, count the number of recalled samples and the number of mispredicted samples in the recalled sample set under a candidate recall threshold, and determine the upper confidence bound of the error rate corresponding to a candidate recall threshold based on the number of recalled samples, the number of mispredicted samples and the preset significance level by inverting the binomial cumulative distribution function.

[0089] The preset significance level δ is a pre-set statistical risk control parameter used to control the confidence level of statistical assurance. For example, it requires that the error rate constraint be met with a probability of at least 1-δ.

[0090] In this embodiment, to ensure the stability of statistical estimation, the candidate recall threshold set is constrained by constructing an equidistant grid recall threshold set in the [0,1] interval. For example, [0,1] is divided into 5000 candidate points, resulting in a candidate recall threshold set containing 5000 candidate recall thresholds. Each candidate recall threshold in the candidate recall threshold set belongs to [0,1]. To recall as many samples as possible while satisfying the error rate constraint, this embodiment adopts a fixed sequence test method from high to low to determine the upper confidence bound of the error rate corresponding to each candidate recall threshold. Specifically, starting from the maximum value of the candidate recall threshold set, the threshold is gradually reduced to find the minimum candidate recall threshold that satisfies the risk constraint.

[0091] Optionally, in this embodiment of the application, a possible implementation is provided for counting the number of recalled samples and the number of incorrectly predicted samples in the recalled sample set based on the output of the target preference evaluation model on the calibration sample set. Specifically, the following operations are performed: S220: Input each calibration sample in the calibration sample set into the target preference evaluation model to obtain the preference prediction result and preference prediction probability of each calibration sample in the calibration sample set.

[0092] Optionally, in this embodiment of the application, when inputting each calibration sample in the calibration sample set into the target preference evaluation model to obtain the preference prediction result and preference prediction probability of each calibration sample in the calibration sample set, the following operations are performed for each calibration sample in the calibration sample set: SB1: Construct a sub-calibration sample group of a calibration sample.

[0093] The sub-calibration sample group includes a first sub-calibration sample and a second sub-calibration sample. The two samples in a calibration sample have different positional relationships in the first and second sub-calibration samples.

[0094] In this embodiment, the sample response at the first input position of the first sub-calibration sample is the first sample response, and the sample response at the second input position of the first sub-calibration sample is the second sample response; the sample response at the first input position of the second sub-calibration sample is the second sample response, and the sample response at the second input position of the second sub-calibration sample is the first sample response.

[0095] SB2: Input the first and second sub-calibration samples into the target preference evaluation model to obtain the prediction probabilities of the third and fourth sub-preferences corresponding to the first sub-calibration sample, and the prediction probabilities of the fifth and sixth sub-preferences corresponding to the second sub-calibration sample.

[0096] The third sub-preference prediction probability is the preference probability of the answer to the sample at the first input position of the first sub-calibration sample; the fourth sub-preference prediction probability is the preference probability of the answer to the sample at the second input position of the first sub-training sample; the fifth sub-preference prediction probability is the preference probability of the answer to the sample at the first input position of the second sub-calibration sample; and the sixth sub-preference prediction probability is the preference probability of the answer to the sample at the second input position of the second sub-training sample. The sum of the third and fourth sub-preference prediction probabilities is 1, and the sum of the fifth and sixth sub-preference prediction probabilities is 1.

[0097] In this embodiment, the first sub-calibration sample is input into the target preference evaluation model to obtain the third and fourth sub-preference prediction probabilities corresponding to the first sub-calibration sample, and the second sub-calibration sample is input into the target preference evaluation model to obtain the fifth and sixth sub-preference prediction probabilities corresponding to the second sub-calibration sample.

[0098] SB3: Average the third and sixth sub-preference prediction probabilities to obtain the first preference prediction probability corresponding to the first sample answer in a calibration sample, and average the fourth and fifth sub-preference prediction probabilities to obtain the second preference prediction probability corresponding to the second sample answer in a calibration sample.

[0099] For example, assuming the prediction probability of the third sub-preference is 0.7, the prediction probability of the fourth sub-preference is 0.3, the prediction probability of the fifth sub-preference is 0.2, and the prediction probability of the sixth sub-preference is 0.8, then the prediction probability of the first preference corresponding to the first sample answer in the calibration sample is (0.7+0.8) / 2=0.75, and the prediction probability of the second preference corresponding to the second sample answer in the calibration sample is (0.3+0.2) / 2=0.25.

[0100] SB4: Compare the first preference prediction probability corresponding to the first sample answer and the second preference prediction probability corresponding to the second sample answer to obtain the preference prediction result and preference prediction probability of a calibration sample.

[0101] In this embodiment of the application, the prediction probability with the largest value among the first preference prediction probability corresponding to the first sample answer and the second preference prediction probability corresponding to the second sample answer is taken as the preference prediction probability of the calibration sample, and the sample answer corresponding to the preference prediction probability of the calibration sample is taken as the preference prediction result. That is, the sample answer with the largest preference prediction probability among the first sample answer and the second sample answer is taken as the preference prediction result.

[0102] For example, assuming the first preference prediction probability is 0.75 and the second preference prediction probability is 0.75, then the preference prediction probability of the calibration sample is 0.75, and the preference prediction result of the calibration sample is that it prefers the first sample answer, that is, the first sample answer is better than the second sample answer.

[0103] In this way, the average preference prediction probability of "position-swapping secondary inference" is used as the final preference prediction probability to reduce the instability caused by the answer order bias, improve the accuracy of preference prediction results and preference prediction probabilities, and make the upper confidence bound of the error rate calculated based on the calibration set and the final target recall threshold more reliable and accurate.

[0104] S221: From the calibration sample set, select each calibration sample whose preference prediction probability is greater than or equal to a candidate recall threshold, and use it as the recalled sample set, and count the number of recalled samples in the recalled sample set.

[0105] In this embodiment of the application, for each calibration sample in the calibration sample set, the following operations are performed: determine whether the preference prediction probability of a calibration sample is greater than or equal to the current candidate recall threshold. If so, the calibration sample is used as a recalled sample and added to the recalled sample set. Then, the number of samples in the recalled sample set is counted to obtain the number of recalled samples.

[0106] In this embodiment of the application, the expression for the number of recalled samples can be:

[0107] Where λ is the current candidate recall threshold. D is the probability of preference prediction for a calibration sample. cal Let x be a calibration sample set.

[0108] S222: If the number of recalled samples is greater than or equal to the preset sample coverage threshold, then the number of incorrectly predicted samples in the recalled sample set is determined based on the preference prediction results and true preference labels of each calibration sample in the recalled sample set.

[0109] In this embodiment, the preset sample coverage threshold is determined by the preset minimum coverage ratio and the number of each calibration sample in the calibration sample set. Specifically, the formula for calculating the preset sample coverage threshold is: ρ·|D cal |, where ρ is the preset minimum coverage ratio (e.g., 0.05, which can be adjusted according to business needs), |D cal | represents the number of each calibration sample in the calibration sample set.

[0110] In this embodiment, when the number of recalled samples is greater than or equal to a preset sample coverage threshold, for each calibration sample in the recalled sample set, the following operations are performed: It is determined whether the preference prediction result of a calibration sample is equal to the true preference label. If so, the calibration sample is determined to be a prediction error sample and added to the prediction error sample set. Then, the number of samples in the prediction error sample set is counted to obtain the number of prediction error samples. When the number of recalled samples is less than the preset sample coverage threshold, the current candidate recall threshold is removed.

[0111] For example, suppose that the preference prediction result of a calibration sample in the recalled sample set is that sample answer 1 is better than sample answer 2, and the true preference label of the calibration sample is that sample answer 2 is better than sample answer 1, then the calibration sample is determined to be a mispredicted sample.

[0112] In this embodiment of the application, the expression for predicting the number of erroneous samples can be:

[0113] Where λ is the current candidate recall threshold. D is the probability of preference prediction for a calibration sample. cal Let x be a calibration sample set. Let y(x) be the preference prediction result for a calibration sample, and y(x) be the true preference label for a calibration sample.

[0114] This introduces a sample coverage threshold as a protection mechanism. When performing statistical inference, if the number of recalled samples is too small (e.g., below the preset minimum coverage ratio, such as 5%), the upper confidence bound of the error rate calculated based on the binomial distribution will be very broad or unstable, leading to unreliable statistical conclusions. By setting a threshold for candidate recall filtering, the effectiveness and robustness of subsequent statistical tests are ensured, avoiding the generation of erroneous or misleading threshold values ​​when the sample size is insufficient, thereby improving the robustness of the entire adaptive threshold solution process.

[0115] Optionally, in this embodiment of the application, a possible implementation is provided for determining the upper confidence bound of the error rate corresponding to a candidate recall threshold based on the number of recalled samples, the number of predicted incorrect samples, and a preset significance level by inverting the cumulative distribution function of the binomial distribution. Specifically, the following operations are performed: S223: Based on the number of incorrectly predicted samples and the number of recalled samples, determine the empirical error rate corresponding to a candidate recall threshold.

[0116] In this embodiment, the ratio of the number of predicted erroneous samples to the number of recalled samples is calculated to obtain the empirical error rate (empirical selectivity risk) of the current candidate recall threshold on the calibration sample set.

[0117] S224: By inverting the cumulative distribution function of the binomial distribution, based on the empirical error rate and the preset significance level, determine the upper confidence bound of the error rate corresponding to a candidate recall threshold.

[0118] In this embodiment, the inversion method of the binomial cumulative distribution function is used to obtain the upper confidence bound of the error rate that satisfies the confidence level, such that with a probability of at least 1 - δ (preset significance level), the empirical error rate is less than or equal to the upper confidence bound of the error rate, i.e., P(R(λ)≤R). ＋ R(λ) ≥ 1 - δ, where R(λ) is the empirical error rate of the current candidate recall threshold on the calibration sample set. ＋ δ represents the upper confidence bound of the error rate corresponding to the candidate recall threshold, and δ is the preset significance level.

[0119] Optionally, in engineering implementation, the upper confidence bound of the error rate corresponding to the current candidate recall threshold can be obtained by solving the following equation: BinomCDF(k(λ); n(λ), r). Here, BinomCDF is the cumulative distribution function of the binomial distribution, the unknown r∈(0,1) is the upper confidence bound of the error rate to be determined, k(λ) is the number of incorrectly predicted samples, and n(λ) is the number of recalled samples.

[0120] In this way, even when the sample size is small and the error rate is close to 0 or 1, it can provide a conservative estimate of the coverage probability at least at the preset significance level. This makes the final calculated upper confidence bound of the error rate have a solid mathematical and statistical basis and can strictly meet the user's preset significance level requirements. Given the preset significance level and the preset target error rate, the target recall threshold can be obtained automatically, which satisfies the requirement of "controlled error rate" in a statistical sense.

[0121] S23: Take the smallest candidate recall threshold in the candidate recall threshold set whose corresponding error rate upper confidence boundary is less than or equal to the preset target error rate as the target recall threshold.

[0122] The preset target error rate α is set in advance, for example: the error rate α of the recalled credible preference answers is expected to be no more than 5%.

[0123] In this embodiment of the application, from the candidate recall threshold set, each intermediate recall threshold whose corresponding error rate upper confidence boundary is less than or equal to the preset target error rate is selected, and the intermediate recall threshold with the smallest value among the intermediate recall thresholds is determined as the final target recall threshold.

[0124] S24: From the candidate response pair set, select the set of credible preference response pairs whose preference prediction probability is greater than or equal to the target recall threshold.

[0125] The preference prediction probability is obtained by inputting the corresponding candidate answer pairs into the target preference evaluation model.

[0126] In this embodiment of the application, a candidate answer pair in the candidate answer pair set includes: a question, a first candidate answer, and a second candidate answer, wherein the first candidate answer and the second candidate answer are different. Optionally, the candidate answer pair set can be formed by multiple dialogue models. Specifically, for each question, the following operations are performed: using multiple dialogue models, multiple candidate answers for a question are generated, and the multiple candidate answers are combined in pairs to form multiple candidate answer pairs corresponding to a question.

[0127] In this embodiment of the application, for each candidate answer pair in the candidate answer pair set, the following operations are performed: a candidate answer pair is input into the target preference evaluation model to obtain the preference probability of the sample answer at the first input position and the preference probability of the sample answer at the second input position. The maximum of the preference probabilities of the sample answers at the two input positions is taken as the preference prediction probability of the candidate answer pair. Then, it is determined whether the preference prediction probability of the candidate answer pair is greater than or equal to the target recall threshold. If so, the candidate answer pair is taken as a credible preference answer pair and added to the credible preference answer pair set.

[0128] The preference prediction probability of a candidate answer pair can be obtained by inputting the two candidate answers into the target preference evaluation model once without changing their positions. For example, (question, candidate answer 1, candidate answer 2) can be directly input into the target preference evaluation model to obtain the preference prediction probability of the candidate answers at the two input positions. Alternatively, the preference prediction probability of a candidate answer pair can be obtained by swapping the input positions of the two candidate answers and inputting them into the target preference evaluation model twice. For example, (question, candidate answer 1, candidate answer 2) can be input into the target preference evaluation model to obtain the preference prediction probability of the candidate answers at the two input positions, and then (question, candidate answer 2, candidate answer 1) can be input into the target preference evaluation model to obtain the preference prediction probability of the candidate answers at the two input positions. Finally, based on the two probabilities corresponding to candidate answer 1, the final preference prediction probability of candidate answer 1 is obtained, and based on the two probabilities corresponding to candidate answer 2, the final preference prediction probability of candidate answer 2 is obtained. This embodiment does not impose any limitations on this approach.

[0129] Furthermore, in the embodiments of this application, after obtaining the set of credible preference answer pairs, it is used for reward model training, preference alignment of the dialogue model, or evaluation of the model's quality. The embodiments of this application do not impose any limitations on this.

[0130] See Figure 3 The diagram shown is a logical illustration of determining the target recall threshold in an embodiment of this application, including the following steps: S30: Input the calibration sample set, preset significance level, preset target error rate, preset minimum coverage ratio, and candidate recall threshold set.

[0131] S31: From high to low, traverse each candidate recall threshold in the candidate recall threshold set and perform the following operations respectively: calculate the number of recalled samples for the current candidate recall threshold.

[0132] S32: Determine whether the number of recalled samples is greater than or equal to the product of the preset minimum coverage ratio and the number of samples in the candidate recall threshold set. If yes, execute S34; otherwise, execute S33.

[0133] S33: Remove the current candidate recall threshold.

[0134] S34: Calculate the number of incorrectly predicted samples corresponding to the current candidate recall threshold.

[0135] S35: Calculate the ratio of the number of incorrectly predicted samples to the number of recalled samples to obtain the empirical error rate.

[0136] S36: By inverting the cumulative distribution function of the binomial distribution, find the upper confidence bound of the error rate corresponding to the current candidate recall threshold, such that P(R(λ)≤R) ＋ )≥1-δ.

[0137] Where R(λ) is the empirical error rate of the current candidate recall threshold on the calibration sample set, R ＋ δ represents the upper confidence bound of the error rate corresponding to the candidate recall threshold, and δ is the preset significance level.

[0138] S37: Determine whether the upper confidence boundary of the error rate is less than or equal to the preset target error rate. If yes, execute S38; otherwise, execute S39.

[0139] S38: Record the current candidate recall threshold as a feasible candidate recall threshold, and continue searching towards a lower candidate recall threshold.

[0140] S39: Output the minimum feasible candidate recall threshold.

[0141] Among them, the minimum feasible candidate recall threshold is the target candidate recall threshold.

[0142] Furthermore, based on the same technical concept, embodiments of this application provide an apparatus for constructing trustworthy preference answer pairs, which is used to implement the above-described method flow of embodiments of this application. For example, see [link to relevant documentation]. Figure 4 As shown, the credible preference answer pair construction device 400 may include: an acquisition module 401, a training module 402, a processing module 403, a determination module 404, and a filtering module 405, wherein: Module 401 is used to acquire training sample sets and calibration sample sets; Training module 402 is used to train the preference evaluation model to be trained using the training sample set and through the answer position exchange mechanism to obtain the target preference evaluation model; Processing module 403 is used to perform the following operations for each candidate recall threshold in the candidate recall threshold set: based on the output of the target preference evaluation model to the calibration sample set, count the number of recalled samples and the number of predicted incorrect samples in the recalled sample set under a candidate recall threshold, and determine the upper confidence bound of the error rate corresponding to a candidate recall threshold based on the number of recalled samples, the number of predicted incorrect samples and the preset significance level by inverting the binomial cumulative distribution function; The determination module 404 is used to take the smallest candidate recall threshold in the candidate recall threshold set whose corresponding error rate upper confidence boundary is less than or equal to the preset target error rate as the target recall threshold. The filtering module 405 is used to filter out a set of credible preference answer pairs from the candidate answer pair set whose preference prediction probability is greater than or equal to the target recall threshold. The preference prediction probability is obtained by inputting the corresponding candidate answer pair into the target preference evaluation model.

[0143] In an optional embodiment, each training sample in the training sample set contains a sample-response pair and a first true preference label, the first true preference label being used to indicate the preferred sample response among the two sample responses contained in the sample-response pair. Each calibration sample in the calibration sample set contains a sample-response pair and a second true preference label, the second true preference label being used to indicate the preferred sample response among the two sample responses contained in the sample-response pair; The first true preference label is obtained based on a preference rating label, and the second true preference label is obtained based on at least two preference rating labels.

[0144] In an optional embodiment, when acquiring the training sample set and the calibration sample set, the acquisition module 401 is further configured to: For each sample question, perform the following operations: Use multiple dialogue models to generate multiple sample answers for a sample question, and combine the multiple sample answers in pairs to form multiple sample answer pairs corresponding to a sample question. Each sample answer pair includes: a sample question, a first sample answer, and a second sample answer. The first sample answer and the second sample answer are different. Each sample answer pair is labeled with a preference score to obtain a labeled dataset, and the standard dataset is divided into a training sample set and a calibration sample set.

[0145] In an optional embodiment, when training a training sample set and training the preference evaluation model to be trained through a response position swapping mechanism to obtain the target preference evaluation model, the training module 402 is further used for: The preference evaluation model to be trained is iteratively trained based on the training sample set to obtain the target preference evaluation model. During one iteration of training, the following operations are performed: Construct a sub-training sample group of the selected target training sample; wherein, the sub-training sample group includes: a first sub-training sample and a second sub-training sample, and the two samples in the target training sample have different positional relationships in the first sub-training sample and the second sub-training sample; The first sub-training sample and the second sub-training sample are respectively input into the preference evaluation model to be trained to obtain the first sub-preference prediction probability corresponding to the first sub-training sample and the second sub-preference prediction probability corresponding to the second training sample. Based on the first sub-preference prediction probability and the second sub-preference prediction probability, the basic cross-entropy loss and mean squared error loss are obtained, and the comprehensive loss is determined based on the basic cross-entropy loss and the mean squared error loss. The first sub-preference prediction probability is the preference probability of the sample answer at the first input position of the first sub-training sample, and the second sub-preference prediction probability is the preference probability of the sample answer at the second input position of the second sub-training sample. Based on the comprehensive loss, the model parameters of the preference evaluation model to be trained are adjusted.

[0146] In one optional embodiment, the sample answer at the first input position of the first sub-training sample is a preferred sample answer, and the sample answer at the second input position of the first sub-training sample is a non-preferred sample answer. The sample answer at the first input position of the second sub-training sample is a non-preference sample answer, and the sample answer at the second input position of the second sub-training sample is a preference sample answer; The first input position is before the second input position, and the preferred sample answer is the preferred sample answer among the two sample answers indicated by the first true preference label in the target training sample.

[0147] In an optional embodiment, when calculating the number of recalled samples and the number of incorrectly predicted samples in the recalled sample set based on the output of the target preference evaluation model to the calibration sample set, the processing module 403 is further configured to: Each calibration sample in the calibration sample set is input into the target preference evaluation model to obtain the preference prediction result and preference prediction probability for each calibration sample in the calibration sample set. From the calibration sample set, select each calibration sample whose preference prediction probability is greater than or equal to a candidate recall threshold, and use it as the recalled sample set, and count the number of recalled samples in the recalled sample set. If the number of recalled samples is greater than or equal to the preset sample coverage threshold, the number of incorrectly predicted samples in the recalled sample set is determined based on the preference prediction results and true preference labels of each calibration sample in the recalled sample set.

[0148] In an optional embodiment, when inputting each calibration sample in the calibration sample set into the target preference evaluation model to obtain the preference prediction result and preference prediction probability of each calibration sample in the calibration sample set, the processing module 403 is further configured to: For each calibration sample in the calibration sample set, perform the following operations: Construct a sub-calibration sample group for a calibration sample; wherein the sub-calibration sample group includes: a first sub-calibration sample and a second sub-calibration sample, and the two samples in a calibration sample have different positional relationships in the first and second sub-calibration samples; The first and second sub-calibration samples are input into the target preference evaluation model to obtain the third and fourth sub-preference prediction probabilities corresponding to the first sub-calibration sample, and the fifth and sixth sub-preference prediction probabilities corresponding to the second sub-calibration sample. The third sub-preference prediction probability is the preference probability of the sample answer at the first input position of the first sub-calibration sample, the fourth sub-preference prediction probability is the preference probability of the sample answer at the second input position of the first sub-training sample, the fifth sub-preference prediction probability is the preference probability of the sample answer at the first input position of the second sub-calibration sample, and the sixth sub-preference prediction probability is the preference probability of the sample answer at the second input position of the second sub-training sample. The first preference prediction probability is obtained by averaging the third and sixth sub-preference prediction probabilities in a calibration sample, and the second preference prediction probability is obtained by averaging the fourth and fifth sub-preference prediction probabilities in a calibration sample, and the second preference prediction probability is obtained by averaging the second and fifth sub-preference prediction probabilities in a calibration sample. The first preference prediction probability corresponding to the first sample answer and the second preference prediction probability corresponding to the second sample answer are compared to obtain the preference prediction result and preference prediction probability of a calibration sample.

[0149] In an optional embodiment, when determining the upper confidence bound of the error rate corresponding to a candidate recall threshold based on the number of recalled samples, the number of mispredicted samples, and a preset significance level by inverting the binomial cumulative distribution function, the processing module 403 is further configured to: Based on the number of incorrectly predicted samples and the number of recalled samples, determine the empirical error rate corresponding to a candidate recall threshold; By inverting the cumulative distribution function of the binomial distribution, an upper confidence bound for the error rate corresponding to a candidate recall threshold is determined based on the empirical error rate and the preset significance level. The preset significance level is a preset statistical risk control parameter.

[0150] Based on the description of the method and apparatus embodiments above, an exemplary embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method according to an embodiment of the present invention.

[0151] This application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0152] This application also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0153] See Figure 5 The diagram shown below illustrates the structure of an electronic device 500 that can serve as a server or client in this application, and is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0154] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0155] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information to electronic device 500. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disks and optical discs. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, WiFi devices, worldwide interoperability for microwave access (WiMax) devices, cellular communication devices, and / or the like.

[0156] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the above-described method for constructing trustworthy preference answer pairs can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the above-described method for constructing trustworthy preference answer pairs by any other suitable means (e.g., by means of firmware).

[0157] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0158] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM) or flash memory, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0159] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device, PLD) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0161] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0162] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0163] Furthermore, it should be understood that the above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of this invention are still within the scope of this application.

Claims

1. A method for constructing credible preference response pairs, characterized in that, include: Obtain the training sample set and calibration sample set; Using the training sample set, the preference evaluation model to be trained is trained through a response position exchange mechanism to obtain the target preference evaluation model; For each candidate recall threshold in the candidate recall threshold set, the following operations are performed: based on the output of the target preference evaluation model to the calibration sample set, the number of recalled samples and the number of incorrectly predicted samples in the recalled sample set are counted under a candidate recall threshold, and the upper confidence bound of the error rate corresponding to the candidate recall threshold is determined based on the number of recalled samples, the number of incorrectly predicted samples and the preset significance level by inverting the binomial cumulative distribution function. The smallest candidate recall threshold in the candidate recall threshold set whose error rate upper confidence boundary is less than or equal to the preset target error rate is taken as the target recall threshold. From the candidate answer pair set, a set of credible preference answer pairs with preference prediction probabilities greater than or equal to the target recall threshold is selected. The preference prediction probability is obtained by inputting the corresponding candidate answer pairs into the target preference evaluation model.

2. The method as described in claim 1, characterized in that, Each training sample in the training sample set contains a sample-response pair and a first true preference label, the first true preference label being used to indicate the preferred sample response among the two sample responses contained in the sample-response pair; Each calibration sample in the calibration sample set contains a sample-response pair and a second true preference label, the second true preference label being used to indicate the preferred sample response among the two sample responses contained in the sample-response pair; The first true preference label is obtained based on a preference rating label, and the second true preference label is obtained based on at least two preference rating labels.

3. The method as described in claim 1, characterized in that, The acquisition of the training sample set and calibration sample set includes: For each sample question, the following operations are performed: multiple dialogue models are used to generate multiple sample answers for a sample question, and the multiple sample answers are combined in pairs to form multiple sample answer pairs corresponding to the sample question. Each sample answer pair includes: the sample question, a first sample answer, and a second sample answer, wherein the first sample answer and the second sample answer are different. Each sample answer pair is labeled with a preference score to obtain a labeled dataset, and the standard dataset is divided into the training sample set and the calibration sample set.

4. The method as described in claim 1, characterized in that, The step of using the training sample set to train the preference evaluation model to be trained through a response position swapping mechanism to obtain the target preference evaluation model includes: The preference evaluation model to be trained is iteratively trained based on the training sample set to obtain the target preference evaluation model. During one iteration of training, the following operations are performed: Construct a sub-training sample group of the selected target training sample; wherein, the sub-training sample group includes: a first sub-training sample and a second sub-training sample, and the two samples in the target training sample have different positional relationships in the first sub-training sample and the second sub-training sample; The first sub-training sample and the second sub-training sample are respectively input into the preference evaluation model to be trained to obtain the first sub-preference prediction probability corresponding to the first sub-training sample and the second sub-preference prediction probability corresponding to the second sub-training sample. Based on the first sub-preference prediction probability and the second sub-preference prediction probability, the basic cross-entropy loss and mean squared error loss are obtained, and the comprehensive loss is determined based on the basic cross-entropy loss and the mean squared error loss. The first sub-preference prediction probability is the preference probability of the sample answer at the first input position of the first sub-training sample, and the second sub-preference prediction probability is the preference probability of the sample answer at the second input position of the second sub-training sample. Based on the comprehensive loss, the model parameters of the preference evaluation model to be trained are adjusted.

5. The method as described in claim 4, characterized in that, The sample answer at the first input position of the first sub-training sample is a preferred sample answer, and the sample answer at the second input position of the first sub-training sample is a non-preferred sample answer; The sample answer at the first input position of the second sub-training sample is a non-preferred sample answer, and the sample answer at the second input position of the second sub-training sample is a preferred sample answer; The first input position is before the second input position, and the preference sample answer is the preferred sample answer among the two sample answers indicated by the first true preference label in the target training sample.

6. The method as described in claim 1, characterized in that, The output of the target preference evaluation model to the calibration sample set, which includes counting the number of recalled samples at a candidate recall threshold and the number of incorrectly predicted samples in the recalled sample set, includes: Each calibration sample in the calibration sample set is input into the target preference evaluation model to obtain the preference prediction result and preference prediction probability for each calibration sample in the calibration sample set. From the calibration sample set, each calibration sample whose preference prediction probability is greater than or equal to a candidate recall threshold is selected as the recalled sample set, and the number of recalled samples in the recalled sample set is counted. If the number of recalled samples is greater than or equal to a preset sample coverage threshold, then the number of incorrectly predicted samples in the recalled sample set is determined based on the preference prediction result and the true preference label of each calibrated sample in the recalled sample set.

7. The method as described in claim 6, characterized in that, The step of inputting each calibration sample in the calibration sample set into the target preference evaluation model to obtain the preference prediction result and preference prediction probability of each calibration sample in the calibration sample set includes: For each calibration sample in the calibration sample set, perform the following operations: Construct a sub-calibration sample group for a calibration sample; wherein the sub-calibration sample group includes: a first sub-calibration sample and a second sub-calibration sample, and the two samples in the calibration sample have different positional relationships in the first sub-calibration sample and the second sub-calibration sample; The first sub-calibration sample and the second sub-calibration sample are respectively input into the target preference evaluation model to obtain the third sub-preference prediction probability and the fourth sub-preference prediction probability corresponding to the first sub-calibration sample, and the fifth sub-preference prediction probability and the sixth sub-preference prediction probability corresponding to the second sub-calibration sample. The third sub-preference prediction probability is the preference probability of the sample answer at the first input position of the first sub-calibration sample, the fourth sub-preference prediction probability is the preference probability of the sample answer at the second input position of the first sub-training sample, the fifth sub-preference prediction probability is the preference probability of the sample answer at the first input position of the second sub-calibration sample, and the sixth sub-preference prediction probability is the preference probability of the sample answer at the second input position of the second sub-training sample. The first preference prediction probability corresponding to the first sample answer in the calibration sample is obtained by averaging the third sub-preference prediction probability and the sixth sub-preference prediction probability; and the second preference prediction probability corresponding to the second sample answer in the calibration sample is obtained by averaging the fourth sub-preference prediction probability and the fifth sub-preference prediction probability. The first preference prediction probability corresponding to the first sample answer and the second preference prediction probability corresponding to the second sample answer are compared to obtain the preference prediction result and preference prediction probability of the calibration sample.

8. The method as described in claim 1, characterized in that, The method of determining the upper confidence bound of the error rate corresponding to the candidate recall threshold based on the number of recalled samples, the number of predicted incorrect samples, and a preset significance level through the inversion of the binomial cumulative distribution function includes: Based on the number of predicted incorrect samples and the number of recalled samples, determine the empirical error rate corresponding to the candidate recall threshold; By inverting the cumulative distribution function of the binomial distribution, based on the empirical error rate and the preset significance level, the upper confidence bound of the error rate corresponding to the candidate recall threshold is determined, wherein the preset significance level is a preset statistical risk control parameter.

9. An apparatus for constructing credible preference response pairs, characterized in that, include: The acquisition module is used to acquire the training sample set and the calibration sample set; The training module is used to train the preference evaluation model to be trained using the training sample set and through the response position exchange mechanism to obtain the target preference evaluation model; The processing module is used to perform the following operations for each candidate recall threshold in the candidate recall threshold set: based on the output of the target preference evaluation model to the calibration sample set, count the number of recalled samples and the number of predicted incorrect samples in the recalled sample set under a candidate recall threshold, and determine the upper confidence bound of the error rate corresponding to the candidate recall threshold based on the number of recalled samples, the number of predicted incorrect samples and the preset significance level by inverting the binomial cumulative distribution function; The determination module is used to select the smallest candidate recall threshold in the candidate recall threshold set whose error rate upper confidence boundary is less than or equal to the preset target error rate as the target recall threshold; The filtering module is used to filter out a set of credible preference answer pairs from the candidate answer pair set whose preference prediction probability is greater than or equal to the target recall threshold. The preference prediction probability is obtained by inputting the corresponding candidate answer pair into the target preference evaluation model.

10. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1-8.

Images