A fact-checking detector construction method and system for medical large model parameterization hallucination

Abstract

The application discloses a fact error detector construction method and system for medical large model parameterization hallucination. The method collects medical question and answer data, and medical experts mark error segments and error causes to construct a structured parameterization hallucination data set. In the large language model supervision fine-tuning process, a multi-task joint optimization mechanism for fact error judgment and error explanation generation is introduced, and classification loss and generation loss are used for joint training to construct a parameterization hallucination detector. The model is evaluated and selected based on indexes such as fact error detection performance, error mode judgment consistency and error segment positioning ability. The parameterization hallucination detector is applied to an online medical question and answer system to perform real-time fact error detection on generated answers and output corresponding error segments and error cause information. The application can effectively improve the safety, reliability and explainability of the medical question and answer system.

Description

Technical Field

[0001] This invention relates to a method and system for constructing a factual error detector for parametric hallucinations in large medical models. Background Technology

[0002] With the rapid development of large-scale model technology, how to better apply large-scale models to various vertical scenarios has become a highly concerned topic. However, the unavoidable illusion problem of large-scale models has become an obstacle in the application process. Currently, in the field of artificial intelligence, large-scale model illusions are generally divided into factual illusions and fidelity illusions. The former mainly refers to inconsistencies with real knowledge, while the latter focuses on inconsistencies with context. In highly rigorous scenarios such as medicine, although both types of illusions can lead to problems, factual illusions are particularly worrying. At best, they may incorrectly answer patients' everyday questions; at worst, they may provide incorrect treatment plans that threaten patient safety.

[0003] To minimize the risk of hallucinations in large models, in addition to improving the quality of the generated content, systematic detection of hallucinations is also necessary. Currently, methods for detecting hallucinations in large models can be divided into two categories: retrieval-based methods, which compare the model output with externally reliable data; and uncertainty-based methods, which analyze inconsistencies in the model's internal signals or behaviors. However, these methods have certain limitations when applied to the medical field. Retrieval-based methods often suffer from the limited scope of externally collected medical knowledge, while uncertainty-based methods lack interpretability and precise diagnostic capabilities.

[0004] Furthermore, most existing hallucination detection methods focus on judging the authenticity of a single generated result, lacking a systematic analysis of stable, repetitive errors in the model. In real-world medical question-answering scenarios, large language models with similar parameter distributions, training paradigms, or infrastructures may consistently generate the same or similar factual errors under different prompts, model versions, or generation conditions. Existing technologies lack effective detection and modeling methods for this type of stable, repetitive error. Summary of the Invention

[0005] To address the aforementioned problems and difficulties in existing technologies, this invention proposes a method and system for constructing a factual error detector for parameterized hallucinations in large medical models.

[0006] This invention first proposes the concept of "parametric illusion," which refers to the phenomenon where large language models with similar parameter distributions, training paradigms, or infrastructure consistently reproduce the same or similar factual error patterns under different cue conditions, model versions, or generation conditions. This type of error originates from inaccurate modeling of medical knowledge in the model's parameter space, exhibiting consistency and reproducibility, unlike random errors in the generation process.

[0007] To address the parametric illusions appearing in large-scale medical models, this invention proposes a method for constructing a factual error detector. By collecting multiple responses generated by relevant large-scale medical models under various prompts and model versions, recurring factual errors are filtered out. Combined with annotations from medical experts, a structured parametric illusion dataset containing "factual error judgment - erroneous fragment - error cause" is constructed. Based on this, a pre-selected large language model is jointly trained on multiple tasks to enable it to identify and interpret stably recurring factual error patterns, thereby constructing a parametric illusion detector.

[0008] The parametric hallucination detector is applied to a medical question-answering scenario to identify factual errors in the question-answering results generated by a large model and output corresponding error explanations. This method can achieve interpretable factual error detection capabilities without relying on external retrieval during the online inference stage, improving the security and reliability of medical large-scale model applications and reducing manual review costs.

[0009] A method for constructing a factual error detector for parameterized hallucinations in large medical models includes the following steps: S1: Collect data and annotate it with expert annotations; S1-1: Collect response data from relevant medical models under multiple prompts and model versions to medical questions, and form them into the form of <question, answer>. S1-2: Filter the data collected in S1-1 to identify stable and recurring factual errors; when the same or similar factual error patterns appear in multiple answers, the corresponding data are marked as parameterized illusion candidate samples; S1-3: Medical experts will annotate the parameterized hallucination candidate samples, including at least three fields: "factual error judgment", "error fragment" and "error reason". Among them, "error fragment" is used to identify the specific content of the answer that contains a factual error, and "error reason" is used to describe the medical knowledge deviation corresponding to the factual error. S2: Construct a dataset based on labeled data for training a parameterized illusion detector; S2-1: Treat parameterized hallucination samples with factual errors as negative samples; S2-2: Select data without factual errors as positive samples; S2-3: Combine the positive and negative samples, and use <question, answer> as the input field of the dataset according to the preset prompt word template, and use "factual error judgment, error fragment and error reason" as the output field to obtain the structured parameterized illusion dataset.

[0010] S3: Divide the dataset constructed in S2 into training and test sets, using medical questions as the grouping unit; S4: Perform multi-task joint training on the pre-selected large language model based on the training set to construct a parameterized illusion detector; S4-1: Calculate the classification loss of the model on the factual error judgment task during training to improve the model's ability to identify factual errors; S4-2: Calculate the model’s generation loss on the error interpretation task during training to improve the model’s ability to explain the causes of factual errors and improve the model’s ability to locate factual errors based on samples containing error fragment annotations. S4-3: The classification loss and generation loss of S4-1 and S4-2 are weighted and averaged and then jointly optimized to obtain the parameterized hallucination detector; S5: Perform multiple training runs and use the test set from S3 to evaluate the parameterized illusion detector in multiple dimensions; S5-1: Evaluate the detection performance of the parameterized illusion detector on the factual error detection task; S5-2: Evaluate the consistency between the decision distribution of the detector output and the expert annotation distribution for the same problem; S5-3: Evaluate the detector's ability to locate erroneous segments; S5-4: Results of comprehensive performance evaluation compared with multiple comparison methods; S5-5: Evaluation of different joint loss weight parameters, thereby determining the weight parameters of the joint training loss in step S4; S6: Based on the evaluation results of S5, select the set of model parameters with the best performance as the final parameterized illusion detector; S7: Integrate the parameterized hallucination detector obtained in S6 into the online medical question-and-answer system to perform factual error detection on the business medical question-and-answer big model, and output the corresponding factual error judgment results, error fragments and error cause information.

[0011] Preferably, the data used in S1 is the response to users' medical questions in real-world scenarios of a large-scale medical question-and-answer model, and it covers multiple departments and fields to ensure the diversity of data distribution.

[0012] Preferably, in S1-1, the number of candidate answers for the same question is greater than 2.

[0013] Preferably, the number of positive and negative samples labeled in S2 is kept relatively balanced.

[0014] Preferably, the classification loss used in S4-1 to calculate the model's performance on the factual error determination task can be formalized as shown in the formula. :

[0015] in, This represents the true label, indicating whether the sample contains factual errors; This represents the probability that the detector predicts a factual error in the sample. This represents the classification loss function value corresponding to the factual error determination task.

[0016] Preferably, the generation loss used in S4-2 to calculate the parameterized hallucination detector in the tasks of error cause generation and error fragment generation can be formalized as shown in the formula. :

[0017] in, The length of the generated text sequence. Represents in a given Predicting the next word in the case of a prefix The probability, This represents the value of the generation loss function used to calculate the generation task.

[0018] Preferably, the joint optimization of classification loss and generation loss in S4-3 can be formalized into a loss function. As in the formula :

[0019] in, This represents the hyperparameter used to balance the classification loss and the generation loss, and is generally set to 0.3. and They represent formulas respectively and formula The corresponding loss function value.

[0020] Preferably, in order to evaluate the performance of the parameterized illusion detector on the detection task on the test set, the prediction confidence of the parameterized illusion detector on "whether there is a factual error" can first be formalized, as shown in the formula. :

[0021] in, These represent the model parameters corresponding to the parameterized hallucination detector. and These respectively represent the input as Under the condition of parameterized hallucination detector corresponding prediction The probability that a variable takes the value 0 or 1. In one embodiment, it can be... The text prefix is ​​constructed to include the prompt "## factual error detection\n". The parameterized illusion detector generates the next token after this prefix, and the token is mapped to either 0 or 1. or .therefore, Indicates that in a given prefix Under the condition, the model determines The normalized confidence score is used to measure the predictive tendency of the model in fact-error detection tasks.

[0022] Based on this, we can define The metrics are used to evaluate the performance of the parameterized illusion detector on the factual error detection task, and are formally represented as the formula. :

[0023] in, This represents the set of positive samples in the test set. For the negative sample set, This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise.

[0024] Preferably, in S5-2, the evaluation of whether the distribution of the parametric hallucination detector output is consistent with the expert annotations can be formalized as follows: :

[0025] in, This represents the set of all evaluation questions. Indicates the number of questions; Indicate the problem The corresponding set of answers, Indicate the problem The corresponding number of answers; For the question The The true label of each answer is used to indicate whether the answer contains a factual error; The model pair problem corresponding to the parameterized hallucination detector The The probability of factual error prediction given by each candidate answer; This is an indicator function; it takes a value of 1 when the condition within the parentheses is true, and a value of 0 otherwise. Specifically, the numerator counts the number of answer pairs where the detector's scoring order matches the true label order among answer pairs with differing true labels; the denominator represents the total number of all answer pairs with different true labels. This indicates the proportion of answer pairs with factually incorrect labeling differences that the detector predicts correctly.

[0026] Preferably, the evaluation of the detector's ability to locate erroneous segments in S5-3 can be formalized as follows:

[0027] Let there be a given answer. The set of all error segments annotated by experts is as follows: The set of erroneous segments located by the parameterized hallucination detector is ,make This represents the similarity obtained based on edit distance normalization, when The time is considered to be the location segment Hit the marked segment , This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Indicates the answer The recall rate of erroneous segments. Further, let be the set of all answers in the test set that contain erroneous segment annotations. Then, as in the formula ,in This indicates the overall error segment localization capability of the parameterized illusion detector on the test set.

[0028] This invention also relates to a factual error detection system for parametric hallucinations in large medical models, comprising: The data acquisition and preprocessing module is used to collect the answers from the medical question-and-answer model and to perform data format unification, deduplication, and cleaning to form the basic data for building a parameterized hallucination detector. The parameterized hallucination sample screening and annotation module is used to analyze the answers generated under multiple model versions, multiple prompt conditions, or multiple generation conditions, screen out the stable and recurring factual error patterns, and organize medical experts to annotate the erroneous segments and causes in the corresponding answers, thereby constructing a parameterized hallucination sample dataset. The prompt word organization module is used to configure prompt word templates for factual error detection and error analysis, and organizes "<question, answer>" into input according to the template, and organizes "factual error detection result, error reason and error fragment" into output; The parameterized hallucination detector building module is used to introduce a joint loss of classification and generation on a pre-selected large language model in the medical field, and train it on the pre-constructed dataset to build a parameterized hallucination detector. The detector performance evaluation and selection module is used to evaluate the factual error detection performance, decision distribution consistency and error segment localization ability of the parameterized illusion detector on the test set, and select the parameterized illusion detector with the best performance based on the evaluation results. The parametric hallucination detector reasoning and integration module is used to integrate the parametric hallucination detector into the medical question-answering system, to perform factual error detection on online answers generated by the business question-answering model and output judgment results and explanation information to support answer interception, manual review and model optimization.

[0029] The present invention also relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method of the present invention.

[0030] The innovation of this invention is: (1) A parameterized hallucination detector modeling paradigm for medical question-and-answer scenarios is proposed. In response to the systematic factual errors that repeatedly occur in multiple versions of large models under different prompting conditions, the concept of "parameterized hallucination" is formally proposed, and a parameterized hallucination detector is constructed based on this concept, so that the model parameters can represent the stable recurrence of error patterns, breaking through the existing methods that only perform hallucination detection for a single answer.

[0031] (2) Construct a structured annotation system to support the training of the parametric hallucination detector. Design a multi-field annotation format that includes "factual error judgment - error fragment - error cause", and complete medical expert annotation on real medical question and answer data to provide a unified data foundation for the parametric hallucination detector to achieve both error judgment and interpretable analysis.

[0032] (3) A multi-task joint optimization training mechanism for constructing a parametric hallucination detector is proposed. By jointly optimizing the factual error judgment task and the error interpretation generation task, the parametric hallucination detector can simultaneously possess the ability to identify factual errors, locate error segments, and explain the causes of errors, thereby improving the interpretability and auditability of the detection results.

[0033] The present invention has the following advantages: (1) Based on the parametric illusion detector, the system can identify recurring factual errors in the model, which are difficult to detect by traditional methods, thereby improving the security and reliability of the medical question answering system.

[0034] (2) The parameterized hallucination detector can generate corresponding error causes and error fragments while outputting the factual error judgment results, so as to realize the interpretable identification of factual errors, which facilitates system iteration optimization and medical compliance review. Compared with the method that only outputs scores or labels, it has stronger interpretability.

[0035] (3) Through the multi-task joint optimization mechanism, the parameterized hallucination detector still has good factual error detection capability with low inference cost, which is suitable for application in large-scale online medical question and answer system and can effectively reduce the cost of manual review. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is the overall flowchart of the method of the present invention.

[0038] Figure 2 This is a structural diagram of the system of the present invention. Detailed Implementation

[0039] Exemplary embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on the present invention are within the scope of protection of the present invention.

[0040] To facilitate understanding of the technical solution of this invention, the relevant terms are explained as follows: In this invention, "parameterized illusion" refers to a factual error pattern that is stably reproduced under different generation conditions by a large language model with similar parameter distribution, training paradigm or infrastructure.

[0041] In this invention, the "parametric hallucination detector" is a functional entity obtained by training a large language model. It is used to determine factual errors in medical question-and-answer results and output the corresponding error reasons and error segments.

[0042] Example 1

[0043] This embodiment provides a method for constructing a factual error detector for parameterized hallucinations in large medical models, including the following steps: S1: Collect data and perform expert annotation on the data. S1-1: Collect response data from a large-scale medical model to medical questions. This study collects response data for medical questions from large language models with similar parameter distributions, training paradigms, or infrastructures from online medical question-answering systems. For the same medical question, multiple answers generated by different model versions, with different prompts, or under different generation conditions are collected and grouped according to the medical question, forming a dataset consisting of a medical question and its corresponding multiple candidate answers.

[0044] S1-2: Screening of parameterized hallucination candidate samples by medical experts

[0045] Medical experts manually compare multiple candidate answers to the same medical question to determine if any of the answers contain factual errors inconsistent with medical common sense, clinical guidelines, expert consensus, or clinical diagnosis and treatment conclusions. When the same or similar factual error patterns consistently appear in multiple answers, the corresponding multiple answers are identified as a parameterized illusion candidate sample group, and the medical question, answer content, model version, and input conditions corresponding to the candidate sample group are recorded. The process then proceeds to steps S1-3 for structured annotation.

[0046] Among them, the same or similar factual error patterns refer to multiple answers that, although the specific wording is different, involve the same medical entities, entity relationships, diagnosis and treatment conclusions, or medical knowledge biases.

[0047] S1-3: Structured annotation of parametric illusion candidate samples

[0048] Medical experts annotated each response in the parametric hallucination candidate sample group, forming a structured annotation result. The annotation includes at least three fields: "Factual Error Detection," "Error Fragment," and "Error Reason." The "Factual Error Detection" field records whether the corresponding response contains a factual error; the "Error Fragment" field identifies the sentence, phrase, or other text content in the response that contains the factual error; and the "Error Reason" field describes the medical knowledge bias corresponding to the error. In cases where multiple factual errors exist in the same response, the error fragment and error reason for each factual error are recorded separately.

[0049] S2: Constructing a dataset based on labeled data

[0050] S2-1: Determine positive and negative samples

[0051] Answers confirmed by medical experts to contain factual errors in steps S1-3 are used as negative samples. Each negative sample includes at least a medical question, the model's answer, the factual error determination, the reason for the error, and the corresponding erroneous segment. For answers containing multiple factual errors, the reason for each error and the erroneous segment are recorded separately. In this embodiment, a total of 2421 answers containing factual errors were selected as negative samples.

[0052] S2-2: Constructing the positive sample set

[0053] From the response data generated by the medical big data model, answers confirmed by medical experts to be free of factual errors were selected as positive samples. For positive samples, the factual error determination was set to "no factual error," and the error reason and error segment fields were set to preset content indicating a correct answer. In this embodiment, a total of 3137 answers without factual errors were selected as positive samples.

[0054] S2-3: Construct the model input and target output according to the prompt word template.

[0055] The negative samples obtained in step S2-1 and the positive samples obtained in step S2-2 are combined. The question field and the answer field are organized into model input according to the preset prompt word template, and the factual error judgment results and the corresponding analysis text are organized into model target output.

[0056] For negative samples, the model's target output includes a judgment label indicating the existence of a factual error, at least one reason for the error, and the corresponding error segment; for positive samples, the model's target output includes a judgment label indicating the absence of a factual error and preset text indicating that the answer is correct.

[0057] The prompt word template is, for example: You are a professor with extensive medical knowledge and clinical experience. Your task is to evaluate whether there are factual errors in the answers provided by the "AI health assistant" to patient questions, analyze the reasons for errors when they are found, and mark the specific erroneous text segments in the answers.

[0058] [Patient Questions]

[0059] {question}

[0060] [AI Answer]

[0061] {Answer}

[0062] Output format requirements

[0063] ## Factual Error Detection

[0064] If the answer contains a factual error, output a label value of 0; if the answer does not contain a factual error, output a label value of 1. Only a single numeric character 0 or 1 is output, without any other text or symbols.

[0065] ##analyze

[0066] If the answer is not factually incorrect, output: "The answer is correct." If the answer contains factual illusions, output at least one reason for the error and the corresponding error fragment, in the following format: Error Reason 1: Briefly explain the medical knowledge bias corresponding to the factual error; Error fragment 1: Quoting verbatim a specific text fragment containing factual errors in the AI's response.

[0067] For negative samples, the target output is, for example: ## Factual Error Detection 0 ##analyze Error reason 1: ... Error fragment 1: ... For positive samples, the target output is, for example: ## Factual Error Detection 1 ##analyze That answer is correct.

[0068] In the above manner, each structured sample is converted into a supervised training sample consisting of the model input text and the target output text, thereby obtaining a dataset for training the parameterized illusion detector.

[0069] S3: Divide the training set and the test set

[0070] The supervised training dataset constructed in step S2 is divided into groups based on medical questions. First, multiple answers corresponding to the same medical question are grouped into the same data group according to the question identifier. Then, each data group is randomly sorted and divided into training and test sets according to a preset ratio, so that all answers corresponding to the same medical question only enter one of the training or test sets, avoiding data of the same medical question appearing in both the training and test sets at the same time.

[0071] In this embodiment, 2274 negative samples are selected from 2421 negative samples as training negative samples, and the remaining 147 are selected as test negative samples; 3000 positive samples are selected from 3137 positive samples as training positive samples, and the remaining 137 are selected as test positive samples. Therefore, the training set includes a total of 5274 samples, including 2274 negative samples and 3000 positive samples; the test set includes a total of 284 samples, including 147 negative samples and 137 positive samples.

[0072] S4: Perform multi-task joint training on the pre-selected large language model based on the training set.

[0073] The training set obtained in step S3 is used to perform supervised fine-tuning of the pre-selected large language model to construct a parameterized illusion detector. For each training sample, the prompt word text constructed in steps S2-3 is used as the model input, and the factual error judgment label and the corresponding analysis text are used as the target output.

[0074] In this embodiment, the Qwen1.5-14B model is used as a pre-selected large language model for supervised fine-tuning.

[0075] S4-1: Calculate the classification loss for the factual error determination task.

[0076] The first tag character following the "## Factual Error Detection" field in the target output is used as the judgment tag position. The judgment tag is 0 or 1, where tag 0 indicates that the corresponding answer contains a factual error, and tag 1 indicates that the corresponding answer does not contain a factual error.

[0077] When the model generates the judgment label, the predicted values ​​of label 0 and label 1 generated by the model at that position are obtained and normalized to obtain the probability that the model predicts that the corresponding answer has a factual error.

[0078] Let the label for determining factual error be... ,in Suppose that the probability of a parameterized hallucination detector predicting a factual error in the corresponding answer is . The classification loss corresponding to the factual error determination task. Represented as:

[0079] By calculating the classification loss separately at the determination label position, the ability of the parameterized illusion detector to determine whether there are factual errors in the answers is improved.

[0080] S4-2: Calculate the generation loss of the model on the error interpretation task.

[0081] Calculate the autoregressive generation loss for the analysis text in the target output, excluding the factual error label.

[0082] For negative samples containing factual errors, the analysis text includes at least the corresponding reason for the error and the erroneous segment. For positive samples without factual errors, the analysis text is a pre-defined text indicating that the answer is correct.

[0083] Specifically, the autoregressive generation loss can be formalized as the formula... :

[0084] in, The length of the generated text sequence. Represents in a given Predicting the next word in the case of a prefix The probability, This represents the loss function used to calculate text quality.

[0085] By calculating the generation loss, the parameterized illusion detector learns to generate error causes corresponding to factual errors, and learns to locate specific error content in the answer when error fragments are labeled.

[0086] S4-3: Joint optimization based on classification loss and generation loss

[0087] The losses obtained in steps S4-1 and S4-2 are weighted and summed according to preset weights to form a joint optimization objective. The parameters of the pre-selected large language model are updated by gradient to achieve a balance between the model's ability to identify factual errors, locate erroneous segments, and explain the causes of errors, thus obtaining a parameterized hallucination detector.

[0088] Specifically, the weighted summation of losses to form a joint optimization objective can be formalized as a loss function. As in the formula :

[0089] in, This represents the weight parameters used to balance the factual error detection task and the error interpretation generation task. In this embodiment, (\alpha) is set to 0.01, 0.05, 0.1, 0.3 and 0.5 respectively. Based on the same training set and training configuration, the pre-selected large language model is subjected to multiple supervised fine-tuning to obtain multiple sets of candidate parameterized hallucination detectors corresponding to different weight parameters.

[0090] Each training group used the AdamW optimizer to update the parameters of the pre-selected large language model using gradients, with an initial learning rate set to... The batch size was set to 8, and a total of 6 training rounds were conducted. During training, the model parameters corresponding to each training round were saved for performance evaluation in step S5.

[0091] S5: Perform a multi-dimensional evaluation of the parameterized illusion detector based on the test set.

[0092] Using the test set obtained in step S3, a multi-dimensional evaluation is performed on each group of candidate parameterized hallucination detectors trained in step S4. The multi-dimensional evaluation includes at least the performance evaluation of factual error detection, the evaluation of the consistency of output distribution under the same medical problem, and the evaluation of the ability to locate erroneous fragments.

[0093] S5-1: Evaluate the performance of the parameterized illusion detector on the factual error detection task.

[0094] On the test set obtained in step S3, the AUC metric is used to measure the detection performance of the parameterized illusion detector in the "presence / absence of factual error" decision task.

[0095] Specifically, we can first formalize the prediction confidence of the parameterized hallucination detector regarding "whether there is a factual error," as shown in the formula. :

[0096] in, These represent the model parameters corresponding to the parameterized hallucination detector. and These represent the probabilities that the model generates a label 0 or 1 at the position of a factual error judgment label, respectively. In this embodiment, it can be... Construct a text prefix containing the prompt phrase "## factual error detection\n".

[0097] Based on this, we can define The metrics are used to evaluate the performance of the parameterized illusion detector on the factual error detection task, and are formally represented as the formula. :

[0098] in, This represents the set of positive samples in the test set. For the negative sample set, This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise.

[0099] At the same time, the accuracy rate is calculated to determine the proportion of factual error judgment labels output by the detector that are consistent with expert annotation labels.

[0100] S5-2: Evaluate the consistency between the output distribution of the parameterized hallucination detector and expert annotations.

[0101] For multiple answers to the same medical question in the test set, compare the consistency between the distribution of factual error judgments given by the parametric hallucination detector and the distribution of expert annotations for different answers, and calculate... Indicators are used to evaluate whether the relative ranking of detectors on the same problem aligns with expert judgment. Specifically, they can be formalized as formulas. :

[0102] in, This represents the set of all evaluation questions. Indicates the number of questions; Indicate the problem The corresponding set of answers, Indicate the problem The corresponding number of answers; For the question The The true label of each answer is used to indicate whether the answer contains a factual error; The model pair problem corresponding to the parameterized hallucination detector The The probability of factual error prediction given by each candidate answer; This is an indicator function; it takes a value of 1 when the condition within the parentheses is true, and a value of 0 otherwise. Specifically, the numerator counts the number of answer pairs where the detector's scoring order matches the true label order among answer pairs with differing true labels; the denominator represents the total number of all answer pairs with different true labels. This indicates the proportion of answer pairs with factually incorrect labeling differences that the detector predicts correctly.

[0103] S5-3: Evaluate the detector's ability to locate erroneous segments.

[0104] For samples with factual errors in the test set, the overlap between the erroneous segments generated by the parametric illusion detector and the expert-annotated segments is compared, and the recall rate of the erroneous segments is calculated to measure the detector's ability to locate factual errors.

[0105] Specifically, the recall rate for evaluating erroneous segments in S5-3 can be formalized as follows:

[0106] Let there be a given answer. The set of all error segments annotated by experts is as follows: The set of erroneous segments located by the parameterized hallucination detector is ,make This represents the similarity obtained based on edit distance normalization, when A value greater than 0.5 indicates a localization segment. Hit the marked segment , This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Indicates the answer The recall rate of erroneous segments. Further, let be the set of all answers in the test set that contain erroneous segment annotations. Then, as in the formula ,in This indicates the overall error segment localization capability of the parameterized illusion detector on the test set.

[0107] S5-4: Comprehensive Performance Evaluation Results

[0108] In this embodiment, the parameterized hallucination detector trained in step S4 is comprehensively evaluated against multiple comparison methods. The comparison methods mainly include detectors using a general large language model with a larger number of parameters and detectors that do not employ multi-task joint loss. The evaluation results are shown in the table below:

[0109] The evaluation results above show that the parameterized hallucination detector trained in this embodiment achieves superior results in terms of accuracy, AUC, consistency of distribution for the same question, and recall of erroneous segments. Specifically, the accuracy reaches 76.7%, the AUC reaches 87.0%, the consistency of distribution for the same question reaches 83.6%, and the recall of erroneous segments reaches 62.7%.

[0110] Compared with detectors that do not employ multi-task joint loss, the parameterized hallucination detector in this embodiment improves AUC, consistency of the same question distribution, and recall of erroneous fragments. This indicates that joint optimization of classification loss and generation loss can simultaneously improve the ability to identify factual errors, the consistency of ranking of different answers to the same question, and the ability to locate erroneous fragments.

[0111] S5-5: Evaluation of different joint loss weight parameters

[0112] To determine the weight parameters of the joint training loss in step S4 In this embodiment, each is set Multiple candidate detectors were obtained with values ​​of 0.01, 0.05, 0.1, 0.3, and 0.5, under the same training set and training configuration, and then evaluated on the test set. The evaluation results are shown in the table below:

[0113] The results above show that when At that time, the candidate parametric hallucination detector achieved an AUC of 87.0% and a false fragment recall of 62.7%, respectively, demonstrating superior overall performance among all candidate detectors. Therefore, in step S6, the final parametric hallucination detector was further determined based on the above evaluation results.

[0114] S6: Determine the final parameterized hallucination detector based on the evaluation results.

[0115] After completing multiple training and evaluations, based on the multi-dimensional evaluation results in step S5, including factual error detection performance, output distribution consistency under the same problem, and error segment localization ability, the parametric illusion detectors corresponding to different model parameters are comprehensively ranked, and the set of model parameters with the best performance is selected as the final parametric illusion detector.

[0116] In this embodiment, when the weight parameter At that time, the candidate parameterized hallucination detector had an AUC of 87.0% and a false fragment recall rate of 62.7%, demonstrating superior overall performance among all candidate detectors. Therefore, it was selected... The corresponding model parameters serve as the final parameterized illusion detector.

[0117] S7: Integrate the parameterized hallucination detector obtained in S6 into the online medical question-and-answer system.

[0118] The system performs real-time factual error detection on the "<question, answer>" generated by the business question-and-answer model, outputting factual error judgments and related error explanations. Different processing strategies can be set based on the detection results output by the parameterized illusion detector, including but not limited to: directly returning the result, refusing to answer, or referring to manual review, thereby improving the overall security and reliability of the system.

[0119] Example 2

[0120] Reference Figure 2 This embodiment provides a factual error detection system for parametric hallucinations in large medical models, the structure of which includes: The data acquisition and preprocessing module is used to collect the answers from the medical question-and-answer model and to perform data format unification, deduplication, and cleaning to form the basic data for building a parameterized hallucination detector. The parameterized hallucination sample screening and annotation module is used to analyze the answers generated under multiple model versions, multiple prompt conditions, or multiple generation conditions, screen out the stable and recurring factual error patterns, and organize medical experts to annotate the erroneous segments and causes in the corresponding answers, thereby constructing a parameterized hallucination sample dataset. The prompt word organization module is used to configure prompt word templates for factual error detection and error analysis, and organizes "<question, answer>" into input according to the template, and organizes "factual error detection result, error reason and error fragment" into output; The parameterized hallucination detector building module is used to introduce a joint loss of classification and generation on a pre-selected large language model in the medical field, and train it on the pre-constructed dataset to build a parameterized hallucination detector. The detector performance evaluation and selection module is used to evaluate the factual error detection performance, decision distribution consistency and error segment localization ability of the parameterized illusion detector on the test set, and select the parameterized illusion detector with the best performance based on the evaluation results. The parametric hallucination detector reasoning and integration module is used to integrate the parametric hallucination detector into the medical question-answering system, to perform factual error detection on online answers generated by the business question-answering model and output judgment results and explanation information to support answer interception, manual review and model optimization.

[0121] Example 3

[0122] The present invention also relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the method for constructing a factual error detector for parameterized hallucinations in a large medical model as described in Embodiment 1.

[0123] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for constructing a factual error detector for parametric hallucinations in large medical models, characterized in that, Includes the following steps: S1: Collect data and annotate it with expert annotations; S1-1: Collect response data from relevant medical models under multiple prompts and model versions to medical questions, and form them into the form of <question, answer>. S1-2: Filter the data collected in S1-1 to identify stable and recurring factual errors; when the same or similar factual error patterns appear in multiple answers, the corresponding data are marked as parameterized illusion candidate samples; S1-3: Medical experts will annotate the parameterized hallucination candidate samples, including at least three fields: "factual error judgment", "error fragment" and "error reason". The "error fragment" is used to identify the specific content of the answer that contains a factual error, and the "error reason" is used to describe the medical knowledge deviation corresponding to the factual error. S2: Construct a dataset based on labeled data for training a parameterized illusion detector; S2-1: Treat parameterized hallucination samples with factual errors as negative samples; S2-2: Select data without factual errors as positive samples; S2-3: Combine the positive and negative samples, and use <question, answer> as the input field of the dataset according to the preset prompt word template, and use "factual error judgment, error fragment and error reason" as the output field to obtain the structured parameterized illusion dataset. S3: Divide the dataset constructed in S2 into training and test sets, using medical questions as the grouping unit; S4: Perform multi-task joint training on the pre-selected large language model based on the training set to construct a parameterized illusion detector; S4-1: Calculate the classification loss of the model on the factual error judgment task during training to improve the model's ability to identify factual errors; S4-2: Calculate the generation loss of the model in the error cause and error fragment generation tasks during training, in order to improve the model's ability to locate and explain factual errors; S4-3: The classification loss and generation loss of S4-1 and S4-2 are weighted and averaged and then jointly optimized to obtain the parameterized hallucination detector; S5: Perform multiple training runs and use the test set from S3 to evaluate the parameterized illusion detector in multiple dimensions; S5-1: Evaluate the detection performance of the parameterized illusion detector on the factual error detection task; S5-2: Evaluate the consistency between the decision distribution of the detector output and the expert annotation distribution for the same problem; S5-3: Evaluate the detector's ability to locate erroneous segments; S5-4: Results of comprehensive performance evaluation compared with multiple comparison methods; S5-5: Evaluation of different joint loss weight parameters, thereby determining the weight parameters of the joint training loss in step S4; S6: Based on the evaluation results of S5, select the set of model parameters with the best performance as the final parameterized illusion detector; S7: Integrate the parameterized hallucination detector obtained in S6 into the online medical question-and-answer system to perform factual error detection on the business medical question-and-answer big model, and output the corresponding hallucination judgment results, erroneous fragments and error cause information.

2. The method for constructing a factual error detector for parametric hallucinations in large medical models according to claim 1, characterized in that, The preset prompt word template in S2-3 includes at least: task description text for describing the factual error detection task, input fields containing "patient questions" and "AI answers", format description text for constraining the output of factual error judgment results, and output constraint text for generating error causes and error fragments.

3. The specific steps of the method for constructing a factual error detector for parameterized hallucinations in a large medical model according to claim 1 are characterized in that: The common optimization objective described in S4 can be formalized as follows: The classification loss used to calculate the model on the factual error determination task can be formalized as follows: : This represents the true label, indicating whether the sample contains factual errors; This represents the probability that the detector predicts a factual error in the sample. This represents the classification loss function value corresponding to the factual error determination task. The generation loss used to calculate the parameterized hallucination detector in the tasks of error cause generation and error fragment generation can be formalized as follows: : The length of the generated text sequence. Represents in a given Predicting the next word in the case of a prefix The probability, This represents the value of the generation loss function used to calculate the generation task. The joint optimization of classification loss and generation loss can be formalized as a loss function. As in the formula : in, This represents the hyperparameter used to balance the classification loss and the generation loss, and is generally set to 0.

3. and They represent formulas respectively and formula The corresponding loss function value.

4. The method for constructing a factual error detector for parameterized hallucinations in large medical models according to claim 1, characterized in that: In step S5, the parametric hallucination detector is evaluated from multiple dimensions by assessing its factual error detection performance, error pattern determination consistency, and error segment localization capability, and the results are quantified using the corresponding evaluation formula.

5. The method according to claim 4, characterized in that, use , , Three metrics are used to evaluate the model's performance. First, the model's prediction confidence on "whether hallucinations exist" is formalized, as shown in the formula. : in, These represent the model parameters corresponding to the parameterized hallucination detector. and These respectively represent the input as Under the condition of parameterized hallucination detector corresponding prediction The probability that a variable takes the value 0 or 1. Therefore, Indicates that in a given prefix Under the condition, the model determines The normalized confidence score is used to measure the predictive tendency of the model in fact-error detection tasks. Based on this, we can define The metrics are used to evaluate the performance of the parameterized illusion detector on the factual error detection task, and are formally represented as the formula. : in, This represents the set of positive samples in the test set. For the negative sample set, This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. To evaluate whether the distribution of the output of the parameterized hallucination detector is consistent with expert annotations, it can be formalized as a formula. : in, This represents the set of all evaluation questions. Indicates the number of questions; Indicate the problem The corresponding set of answers, Indicate the problem The corresponding number of answers; For the question The The true label of each answer is used to indicate whether the answer contains a factual error; For the parameterized hallucination detector to address the problem The The probability of factual error prediction given by each candidate answer; This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Used to evaluate the detector's ability to locate erroneous segments. It can be formalized as follows: Let there be a given answer. The set of all error segments annotated by experts is as follows: The set of erroneous segments located by the parameterized hallucination detector is ,make This represents the similarity obtained based on edit distance normalization, when The predicted fragment is considered at that time. Hit the marked segment , This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Indicates the answer The recall rate of the erroneous segments. Further, let be the set of all answers in the test set that contain erroneous segment annotations. Then, as in the formula ,in This represents the overall error segment localization capability of the parameterized illusion detector on the test set.

6. A factual error detection system for parametric hallucinations in large medical models, characterized in that, include: The data acquisition and preprocessing module is used to collect the answers from the medical question-and-answer model and to perform data format unification, deduplication, and cleaning to form the basic data for building a parameterized hallucination detector. The parameterized hallucination sample screening and annotation module is used to analyze the answers generated under multiple model versions, multiple prompt conditions, or multiple generation conditions, screen out the stable and recurring factual error patterns, and organize medical experts to annotate the erroneous segments and causes in the corresponding answers, thereby constructing a parameterized hallucination sample dataset. The prompt word organization module is used to configure prompt word templates for factual error detection and error analysis, and organizes "<question, answer>" into input and "factual error detection result, error reason and error fragment" into output according to the template; The parameterized hallucination detector building module is used to introduce a joint loss of classification and generation on a pre-selected large language model in the medical field, and train it on the pre-constructed dataset to build a parameterized hallucination detector. The detector performance evaluation and selection module is used to evaluate the factual error detection performance, decision distribution consistency and error segment localization ability of the parameterized illusion detector on the test set, and select the parameterized illusion detector with the best performance based on the evaluation results. The parametric hallucination detector reasoning and integration module is used to integrate the parametric hallucination detector into the medical question-answering system, to perform factual error detection on online answers generated by the business question-answering model and output judgment results and explanation information to support answer interception, manual review and model optimization.

7. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the method of any one of claims 1-6.

Images