A risk control method for cloud edge model collaborative medical knowledge reliable answering

Abstract

The present application relates to a kind of cloud edge model collaborative medical knowledge reliable answer risk control method.Forwarding, including: obtaining medical question and answer sample and constructing calibration dataset and test dataset;Utilize edge model to generate answer and calculate corresponding uncertainty score by uncertainty quantification method;Model level selection indicator variable and error indicator variable and system level selection indicator variable and error indicator variable are constructed;Based on calibration dataset, the experience value of system overall error discovery rate is statistically calculated, and upper confidence limit is constructed;According to the target risk level set, the first threshold and the second threshold are updated jointly, and the final threshold pair is determined according to the rule that system overall accepted problem number is prior, and edge model accepted problem number is second best;In test stage, the final threshold pair is used to execute online question and answer.This method can statistically control the overall error discovery rate of system, improve system overall acceptance rate and edge model processing proportion under the premise of meeting risk constraint.

Description

Technical Field

[0001] This invention belongs to the fields of medical artificial intelligence, medical large-scale models, natural language processing, cloud-edge collaborative computing and intelligent risk control technology, and specifically relates to a risk control method for reliable answers to medical knowledge in cloud-edge model collaborative computing. Background Technology

[0002] With the development of artificial intelligence technology, intelligent question-answering methods based on large language models and multimodal medical models have been gradually applied in scenarios such as medical knowledge consultation, online consultation, medical education, and health management. Medical question-answering models can automatically generate answers based on user-inputted medical questions, providing initial information support for patients or medical administrators, and offering advantages such as improving information acquisition efficiency and alleviating the strain on medical resources. However, medical question-answering scenarios are high-risk applications. If the model outputs incorrect, misleading, or incomplete answers, it may adversely affect subsequent diagnostic judgments, health management decisions, or patient cognition. Therefore, in addition to focusing on the quality of the answers, such systems also need to effectively control the reliability of the output results.

[0003] In practical deployments, medical question-answering models typically exhibit significant differences under varying computing resources and service costs. Models deployed at the edge or on local devices offer advantages such as fast response times, low invocation costs, and privacy protection; however, their answer quality is often limited by model size, training data, knowledge update capabilities, and computing power. Dedicated large-scale medical models deployed in the cloud are usually trained on larger-scale, higher-quality, and continuously updated medical corpora, clinical knowledge bases, or proprietary medical data, resulting in higher answer quality, but at a higher cost and unsuitable for direct use on all questions in real-time interactive scenarios. Therefore, establishing a reasonable hierarchical collaboration mechanism between edge models, cloud models, and human experts to control question-answering risks while minimizing system costs has become a crucial issue in the practical application of medical artificial intelligence.

[0004] In existing technologies, a common approach is to use the confidence level, uncertainty, or scoring function of the model output to filter and route question samples. For example, when the local model generates a relatively certain answer to a medical question, the result is accepted directly; when the local model has high uncertainty, the question is submitted to a more powerful but more expensive cloud model for further answering; if the cloud model still lacks sufficient confidence, it is further handed over to human experts for answering or verification. This type of method can balance system cost and question-answering quality to a certain extent, and therefore has high application value in cloud-edge collaborative inference, selective prediction, and rejection decision-making.

[0005] Meanwhile, recent studies on selective and conformal prediction have shown that a batch of independent calibration data can be used to statistically analyze the model's risk under a given selection rule, and the selection threshold can be calibrated accordingly. This allows for statistical guarantees at a user-specified risk level when the test and calibration data meet conditions such as commutativity or identical distribution. Existing research has also proposed constructing upper confidence bounds based on empirical risk on the calibration set and selecting thresholds through methods such as fixed-sequence tests to achieve provable control over selective risk on a smaller calibration set. Furthermore, related frameworks indicate that in multi-level model cascading, the decision of when to trust a weaker model and when to upgrade to a stronger model can be based on confidence levels, thereby reducing invocation costs while ensuring risk levels.

[0006] However, existing methods still have the following shortcomings. First, many cloud-edge collaborative medical question-answering methods mainly focus on average answer quality, average cost, or empirical routing effects, lacking a mechanism for strictly constraining the risk of the final set of answers accepted by the system, especially lacking system-level control over the error rate, i.e., the proportion of errors in accepted answers. Second, existing research based on selective prediction or conformal calibration mostly focuses on single-model selective acceptance or provides general descriptions of cascading frameworks, without establishing a unified system-level acceptance indication, error indication, and risk statistical modeling mechanism for the specific process of "edge-end answering—cloud upgrade—expert backup" in medical question-answering scenarios. Therefore, it is difficult to directly use it to accurately calibrate the error rate of the entire cloud-edge collaborative system. Third, there is a coupling relationship between edge thresholds and cloud thresholds: edge thresholds not only determine which questions are answered directly by the local model, but also determine which questions will flow to the cloud, thus affecting the question distribution and overall system risk in the cloud stage; if two thresholds are set independently, it is often impossible to guarantee that the overall risk at the system level meets the preset requirements.

[0007] Furthermore, in medical question-and-answer scenarios, simply pursuing strict risk control is insufficient. Setting overly conservative thresholds, while reducing the risk of errors, can lead to a large number of questions being sent to the cloud or even further to human experts, significantly increasing costs and reducing the system's automated processing capabilities. Therefore, it is also necessary to maximize the system's acceptance rate of question samples while meeting risk control requirements, in order to achieve a balance between question-and-answer reliability and resource consumption.

[0008] Therefore, there is an urgent need for a cloud-edge collaborative answering and risk control method for medical question-answering scenarios. This method should be able to construct a statistical upper bound on the overall error detection rate of the system based on the uncertainty information of the edge model and the cloud model, and jointly calibrate multiple thresholds. This would enable the system to accept as many questions as possible while meeting the risk level specified by the user, thereby balancing the security, efficiency and service cost of medical question-answering. Summary of the Invention

[0009] This invention aims to overcome the shortcomings of existing cloud-edge collaborative medical question-answering methods, which lack a system-level statistical risk control mechanism.

[0010] To achieve the above-mentioned technical objectives and effects, the present invention provides the following technical solution:

[0011] A risk control method for reliable answering of medical knowledge in cloud-edge model collaboration includes the following steps:

[0012] S1: Obtain a medical question and answer sample set, and divide the sample set into a calibration dataset for statistical risk upper bound estimation and threshold calibration and a test dataset for deployment phase verification. Each question sample in the sample set is associated with at least one of the following: reference answer, standard answer, expert answer, or annotation information used to judge the correctness of the answer.

[0013] S2: For any input medical question, the edge model first generates a corresponding answer and uses a preset uncertainty quantification method to obtain the uncertainty score of the edge model. If the score does not exceed the first threshold, the edge answer is accepted directly; otherwise, it is routed to the cloud model. The cloud model generates a corresponding answer and uses the same uncertainty quantification method to obtain the cloud uncertainty score. If the score does not exceed the second threshold, the cloud answer is accepted; otherwise, the question is submitted to the expert review module.

[0014] S3: Construct model-level selection indicator variables and model-level error indicator variables to represent whether the question is accepted by the corresponding model and whether the answer generated by the corresponding model is wrong, respectively;

[0015] S4: Based on the model-level indicator variables, construct system-level selection indicator variables and system-level error indicator variables to represent whether the problem is accepted by the entire cloud-edge collaborative system and whether an error occurs when the problem is accepted by the system, respectively.

[0016] S5: Given the first threshold and the second threshold, based on the system-level selection indicator variable and the system-level error indicator variable of each problem sample in the calibration dataset, count the number of problems accepted by the system and the number of problems accepted by the system but erroneous, and calculate the empirical statistic of the overall error detection rate of the system.

[0017] S6: Based on the empirical error detection rate of the calibration dataset, construct an upper confidence bound for the overall error detection rate of the system, wherein the upper confidence bound meets a preset confidence probability requirement;

[0018] S7: Based on the target risk level set by the user, calculate the upper confidence bound of the overall system error detection rate for each of the multiple candidate first and second threshold combinations, and select a set of candidate threshold pairs whose upper confidence bound does not exceed the target risk level.

[0019] S8: Among the candidate threshold pairs that meet the risk constraints, the final threshold pair is determined according to the following priority order: the first optimization objective is to maximize the number of problem samples accepted by the system as a whole, and the second optimization objective is to maximize the number of problem samples accepted by the edge model when the number of samples accepted by the system as a whole is the same.

[0020] S9: Medical questions are processed sequentially through edge models and cloud models, and the final threshold is used to determine whether to accept the corresponding model's answer. Questions that are not accepted are submitted to the expert review module.

[0021] Specifically, in step S1, the calibration dataset and the test dataset satisfy the exchangeability condition or the independent and identically distributed condition.

[0022] Specifically, in step S2, the uncertainty component method is used to output the corresponding uncertainty score based on the model's response to the input medical question. The larger the uncertainty score, the more uncertain the response is, and the smaller the score, the more certain the response is.

[0023] Specifically, the uncertainty quantification method uses the prediction entropy calculated from the category vector obtained based on the logit information of the last layer of the large language model. For a multiple-choice medical question-answering task, the uncertainty score is defined as:

[0024]

[0025] in, Indicates the number of candidate options. Represented as model for the th The first sample The predicted probability of each option.

[0026] Specifically, in step S3:

[0027] The selection indicator variable for the edge model is defined as:

[0028]

[0029] in, For the first Marginal uncertainty score of each sample The first threshold, Indicates an indicator function;

[0030] The error indicator variable for the edge model is defined as follows:

[0031]

[0032] in, For the edge model, the first The answers of one sample For the first The standard answer for each sample This indicates that the answer is inconsistent with the standard answer;

[0033] The selection indicator variable for the cloud model is defined as:

[0034]

[0035] in, For the first Cloud uncertainty score for each sample The second threshold;

[0036] The error indicator variable for the cloud model is defined as:

[0037]

[0038] in, For the edge model, the first The responses of one sample;

[0039] Specifically, in step S4:

[0040] The system-level selection indicator variable is obtained by combining the edge model selection indicator variable and the cloud model selection indicator variable, and is defined as follows:

[0041]

[0042] The system-level error indicator variable is obtained by combining the model selection indicator variable and the corresponding model's prediction error indicator variable, and is defined as follows:

[0043]

[0044] Specifically, in step S5, the overall system error detection rate is defined as the ratio of the number of problems accepted by the system that resulted in errors to the total number of problems accepted by the system:

[0045]

[0046] in The total number of samples in the calibration dataset.

[0047] Specifically, in step S6, the upper confidence bound of the overall system error detection rate is denoted as... The upper confidence bound satisfies:

[0048]

[0049] in, Given a first threshold Second threshold The overall error detection rate of the system below. This is a confidence parameter.

[0050] Specifically, in step S6, the Clopper-Pearson method is used to construct an upper confidence bound for the overall error detection rate of the system. This upper bound is determined when the system accepts a certain number of problems. At that time, the upper confidence bound is:

[0051]

[0052] in Let be the quantile function of the Beta distribution. The number of problems accepted by the system and that resulted in errors; when When the threshold is reached, the corresponding threshold pair is considered an infeasible solution.

[0053] Specifically, in step S7, the candidate first threshold set consists of the uncertainty scores of all edge models in the calibration dataset, and the candidate second threshold set consists of the uncertainty scores of all cloud models in the calibration dataset.

[0054] Compared with the prior art, the beneficial effects of the present invention are:

[0055] 1. Achieve system-level rather than single-model-level risk control. This invention does not rely solely on empirical screening of individual model outputs, but rather constructs unified system-level selection indicator variables and system-level error indicator variables around edge and cloud models, and based on these, uniformly models and controls the error detection rate of the final accepted response results of the entire cloud-edge collaborative system.

[0056] 2. Explicit statistical guarantees. This invention utilizes statistical regularities observed on independent calibration datasets to construct an upper confidence bound for the overall system error detection rate. Furthermore, when the calibration and test data satisfy the conditions of commutativity or independent identical distribution, the overall system error detection rate during the testing phase can meet the user-defined target risk level requirements, thereby improving the reliability of the medical question-and-answer system in high-risk scenarios.

[0057] 3. Ensuring system effectiveness through joint threshold calibration. This invention employs a joint update mechanism for edge and cloud thresholds, rather than calibrating the two models independently. This accurately reflects the flow of problems in the cloud-edge collaborative link, avoiding disruption of overall system risk control due to local calibration failures.

[0058] 4. Maximize automated processing capabilities while keeping risks under control. This invention takes "maximizing the number of questions accepted by the system as the primary optimization objective" and "maximizing the number of questions accepted by the edge model as the secondary optimization objective." Under the premise of controlled risks, it retains as many questions as possible for automated processing and prioritizes retaining more questions processed by the edge model when the overall system acceptance capacity is the same, thereby reducing cloud call costs and improving question-and-answer efficiency.

[0059] 5. Balancing security, cost, and practical deployment. This invention establishes a three-tiered collaborative mechanism: "edge model—cloud model—expert review." When the question is relatively easy, the edge model provides a quick answer; when the question is more complex or the model has higher uncertainty, it is escalated to the cloud model; when the cloud model still does not meet the risk requirements, it is then handed over to experts for answering or review. This structure can balance the security of medical question answering, the cost of cloud access, and the actual deployment efficiency of the system, and has strong engineering practical value. Attached Figure Description

[0060] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the specific embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart of the cloud-edge collaborative answering and risk control method in medical question answering in an embodiment of the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] This invention provides a cloud-edge collaborative answering and risk control method for medical question answering tasks. The system includes at least an edge model, a cloud model, and an expert review module. The edge model is a low-cost local model, and the cloud model is a high-performance, high-cost model. The method includes the following steps:

[0064] S1: Obtain medical question-and-answer samples and construct a dataset. Obtain a medical question-and-answer sample set, wherein each question sample in the sample set is associated with at least one of the following: a reference answer, a standard answer, an expert answer, or annotation information used to judge the correctness of the answer. Divide the sample set into a calibration dataset. and test dataset ,in To calibrate the number of samples in the dataset, This represents the number of samples in the test dataset. The calibration dataset is used for statistical risk upper bound estimation and threshold calibration, while the test dataset is used for validation during the deployment phase.

[0065] Preferably, the calibration dataset and the test dataset satisfy the exchangeability condition or the independent and identically distributed condition, so as to provide statistical guarantees for the threshold obtained by using the calibration dataset during the testing phase.

[0066] S2: Generates answers using edge and cloud models and performs hierarchical routing based on uncertainty quantification results. For any input medical question... First, from the edge model Generate corresponding answers And adopt a pre-defined uncertainty quantification method. The answers generated by the edge model are quantified to obtain the uncertainty score of the edge model. When the uncertainty score of the edge model does not exceed the first threshold When the uncertainty score of the edge model is higher than the first threshold, the system directly accepts the answer from the cloud model; when the uncertainty score of the edge model is higher than the first threshold, the system routes the question to the cloud model. The cloud-based model receives the question and generates a corresponding answer. And employ the aforementioned uncertainty quantification method The answers generated by the cloud-based model are quantified to obtain the uncertainty score of the cloud-based model. When the uncertainty score of the cloud model does not exceed the second threshold. When the uncertainty score of the cloud model exceeds a second threshold, the system accepts the answer from the cloud model; when the uncertainty score of the cloud model exceeds a second threshold, the system refuses to automatically output the answer and submits the question to a human expert for answering or review. The uncertainty quantification method mentioned above... This is used to output the corresponding uncertainty score based on the model's response to the input medical question, so as to characterize the reliability of the model's response; preferably, the larger the uncertainty score, the more uncertain the response, and the smaller the uncertainty score, the more certain the response.

[0067] S3: Construct model-level selection indicator variables and model-level error indicator variables. For any problem sample, define the selection indicator variable for the edge model. This is used to indicate whether the problem is directly accepted by the system from the edge model; it defines the error indicator variable for the edge model. This is used to indicate whether the answer generated by the edge model is incorrect.

[0068] Furthermore, define the selection indicator variable for the cloud model. This indicates whether the problem was accepted by the system from the cloud model, provided that the edge model was not accepted; it defines the error indicator variable for the cloud model. This is used to indicate whether the answer generated by the cloud-based model is incorrect.

[0069] Whether an answer is incorrect can be determined based on the reference answer, standard answer, expert annotation results, semantic matching judgment results, or preset correctness judgment rules; This indicates that the answer differs from the reference answer, depending on the specific question type and user requirements; This indicates an indicator function that takes the value 1 when the condition within the parentheses is true, and 0 otherwise.

[0070] S4: Construct system-level selection and error indicator variables. Based on the model-level indicator variables, construct a system-level selection indicator variable to indicate whether the problem is accepted by the entire cloud-edge collaborative system. Since any problem will be accepted by at most one model (either the edge model or the cloud model), the system-level selection indicator variable is a binary variable, which can be obtained by combining the edge model selection indicator variable and the cloud model selection indicator variable, defined as follows:

[0071] .

[0072] Furthermore, a system-level error indicator variable is constructed and defined as follows:

[0073] ,

[0074] This indicates whether an error occurred when the question was accepted by the entire system. The system-level error indicator variable is set to 1 when the edge model is accepted and the edge model responds with an error; the system-level error indicator variable is set to 1 when the cloud model is accepted and the cloud model responds with an error; otherwise, the system-level error indicator variable is set to 0.

[0075] S5: Construct an empirical statistic for the overall system error detection rate. Given a first threshold and a second threshold, based on the system-level selection indicator variable and the system-level error indicator variable for each problem sample in the calibration dataset, count the number of problems accepted by the system and the number of problems accepted but erroneous, and calculate an empirical statistic for the overall system error detection rate. Preferably, the overall system error detection rate can be expressed as the ratio of "the number of problems accepted but erroneous" to "the total number of problems accepted by the system," defined as:

[0076] .

[0077] S6: Based on the system empirical error detection rate observed in the calibration dataset, construct an upper confidence bound for the overall system error detection rate. Preferably, the upper confidence bound adopts the form of an upper confidence bound within a confidence interval, defined as follows: The upper confidence bound satisfies:

[0078] .

[0079] in, Given a first threshold Second threshold The overall system error detection rate is defined as the probability of an error occurring under acceptable system conditions, i.e.:

[0080] ,

[0081] This represents the confidence parameter or significance level.

[0082] S7: Jointly update threshold pairs under risk constraints. Based on the user-defined target risk level, for multiple candidate first and second threshold combinations, calculate the overall system accepted issues, the number of accepted issues in edge models, and the system empirical error detection rate on the calibration dataset for each threshold pair, and construct an upper confidence bound for the overall system error detection rate. Select a set of candidate threshold pairs that satisfy the condition that "the upper confidence bound of the overall system error detection rate does not exceed the target risk level." The candidate first threshold set can be composed of the uncertainty score set of edge models in the calibration dataset, defined as:

[0083] ,

[0084] The candidate second threshold set can be composed of the set of uncertainty scores of the cloud model in the calibration dataset, defined as:

[0085] .

[0086] S8: Determine the final threshold pair according to the dual-objective criterion. Among all candidate threshold pairs that satisfy the risk constraints, determine the final threshold pair in the following priority order:

[0087] 1. Primary optimization objective: To maximize the number of problem samples accepted by the system as a whole.

[0088] ;

[0089] 2. Second optimization objective: When multiple threshold pairs achieve the maximum number of problem samples accepted by the system as a whole, further select the threshold pair that maximizes the number of problem samples accepted by the edge models.

[0090] .

[0091] In other words, if only one threshold pair results in the system receiving the maximum number of issues, that threshold pair is directly selected. If multiple threshold pairs result in the same maximum number of issues received, the number of issues received by the edge models is further compared among these pairs, and the threshold pair with the highest number of issues received by the edge models is selected as the final threshold pair. The significance of this dual-objective strategy is: under the premise of controllable risk, priority is given to improving the overall automatic processing capability of the system; when the overall system acceptance capability is the same, priority is given to increasing the acceptance volume of weak models, i.e., edge models, thereby reducing the number of cloud calls, lowering costs, and improving deployment efficiency.

[0092] S9: Online Q&A Process During the Testing Phase. During the testing phase, for any unanswered medical question, the edge model first generates an answer and calculates the corresponding uncertainty score using uncertainty quantification method U. If the edge model's uncertainty score does not exceed the final first threshold, the system directly accepts the edge model's answer. If the edge model's uncertainty score is higher than the final first threshold, the question is routed to the cloud model. The cloud model then generates an answer and calculates the corresponding uncertainty score using uncertainty quantification method U. If the cloud model's uncertainty score does not exceed the final second threshold, the system accepts the cloud model's answer. If the cloud model's uncertainty score is still higher than the final second threshold, the system refuses to automatically output the answer and submits the question to experts for answering or review.

[0093] Through the above process, this invention achieves statistical control of the overall error detection rate of the cloud-edge collaborative medical question-answering system, and maximizes the overall acceptance rate of the system while meeting risk constraints, while prioritizing the retention of more questions that are directly processed by the edge model.

[0094] The invention will now be further described with reference to the accompanying drawings.

[0095] This embodiment focuses on medical question-answering tasks, using the MedMCQA dataset to validate the cloud-edge collaborative answering and risk control method for medical question answering proposed in this invention. The MedMCQA dataset is a multiple-choice question-answering dataset in the medical field, where each question corresponds to multiple candidate options and a standard answer. In this embodiment, 4.18K samples from the MedMCQA dataset validation set are selected as experimental samples and divided into a calibration dataset and a test dataset according to a 0.5 split ratio. One portion of the samples is used for threshold calibration, and the other portion is used for risk control validation during the testing phase.

[0096] The cloud-edge collaborative system in this embodiment includes an edge model, a cloud model, and an expert review module. The edge model uses the LLaMA-3.1-8B-Instruct model, fine-tuned on the MedMCQA dataset training set, as a low-cost local model for quickly answering input medical questions. The cloud model uses the LLaMA-3.1-72B-Instruct model, trained on the MedMCQA dataset training set using the same fine-tuning method, as a high-performance, high-cost model for further answering questions with high uncertainty from the edge model. The expert review module handles questions where neither the edge model nor the cloud model meets the acceptance criteria.

[0097] For any input medical question First, from the edge model Generate corresponding answers Since this embodiment uses a multiple-choice medical question-answering task, the edge model can output the last layer logit information corresponding to each candidate option, and after normalization, form a category probability vector, denoted as:

[0098] ,

[0099] in, Indicates the number of candidate options. This indicates that the edge model is for the problem sample. For the The predicted probabilities given by each option. The uncertainty quantification method in this embodiment. The prediction entropy is calculated using the category vector obtained from the logit information of the last layer of the large language model. Therefore, the uncertainty score corresponding to the edge model is defined as:

[0100] .

[0101] The larger the prediction entropy, the less certain the edge model's answer to the current medical question; the smaller the prediction entropy, the more certain the edge model's answer to the current medical question.

[0102] Let the acceptance threshold corresponding to the edge model be... When the uncertainty score of the edge model Not exceeding the first threshold When the system directly accepts the response generated by the edge model, the uncertainty score of the edge model is... Above the first threshold If the system deems the edge model's answer to the current question unreliable, it routes the question to the cloud model. .

[0103] For the problem sample of the model routed to the cloud By cloud model Generate corresponding answers It also outputs the last-level logit information for each candidate option, which is then normalized to form a class probability vector:

[0104] ,

[0105] in, This indicates that the cloud model is for the problem sample. For the The predicted probabilities given by each option. Furthermore, the uncertainty score corresponding to the cloud model is defined as:

[0106]

[0107] Let the acceptance threshold corresponding to the cloud model be... When the uncertainty score of the edge model Not exceeding the second threshold When the system receives the response generated by the cloud model, the uncertainty score of the cloud model is... Above the second threshold When this happens, the system refuses to automatically output an answer and submits the current question to the expert review module for manual answering or review.

[0108] To control the error detection rate of the final accepted response results of the entire cloud-edge collaborative system, model-level selection indicator variables and model-level error indicator variables were constructed on the calibration dataset. For any question sample... Define the selection indicator variable for the edge model as:

[0109] ,

[0110] This is used to indicate whether the problem is directly accepted by the system from the edge model.

[0111] Since the MedMCQA dataset is a multiple-choice question-answering task, where each question corresponds to a standard answer, the error indicator variable for the edge model is defined as follows:

[0112] ,

[0113] This indicates whether the problem is accepted by the system from the cloud model, provided that the edge model was not accepted.

[0114] Define the error indicator variable for the cloud model as follows:

[0115] .

[0116] Based on this, construct system-level selection indicator variables:

[0117] ,

[0118] Used to represent a problem sample Whether it is accepted by the entire cloud-edge collaborative system. Since any question will be accepted by at most one model, either the edge model or the cloud model, the system-level selection indicator variable is a binary variable.

[0119] Furthermore, construct system-level error indicator variables:

[0120] ,

[0121] This indicates whether an error occurred when the question was accepted by the system.

[0122] Given a first threshold Second threshold In this case, based on the calibration dataset, the total number of problem samples accepted by the overall statistical system is:

[0123] ,

[0124] And the total number of problem samples accepted and encountered by the system as a whole:

[0125] ,

[0126] in, This represents the total number of problematic samples in the calibration dataset.

[0127] Furthermore, the empirical statistic for the overall system error detection rate is defined as follows:

[0128] .

[0129] when In such cases, the threshold pair can be considered an infeasible solution, or the empirical error detection rate can be processed according to preset rules.

[0130] In order to control the overall error detection rate of the system during the testing phase, this embodiment does not directly use the empirical error detection rate. Instead of selecting a threshold, an upper confidence bound for the overall system error detection rate is constructed based on calibration data. Let the user-specified target risk level be... The confidence parameter is Then the upper confidence bound of the overall system error detection rate is denoted as . It satisfies:

[0131] ,

[0132] in, This represents the overall error detection rate of the system under a given threshold.

[0133] In this embodiment, the upper confidence bound is constructed using the Clopper–Pearson method. When At that time, the Clopper-Pearson upper confidence bound for the overall system error detection rate is defined as:

[0134] ,

[0135] in, express Distribution quantile function. When If the threshold pair is not feasible, it will be considered an infeasible solution and will not be included in the candidate threshold pair set.

[0136] During the calibration phase, candidate first threshold sets and candidate second threshold sets are constructed based on the uncertainty scores of the edge and cloud models in the calibration dataset:

[0137] .

[0138] For each pair of candidate thresholds Calculate the upper confidence bounds for the total number of problem samples accepted by the system as a whole, the number of problem samples accepted by the edge models, the empirical error detection rate, and the overall error detection rate of the system, and retain only candidate threshold pairs that satisfy the following formula:

[0139]

[0140] Among all candidate threshold pairs that satisfy the risk constraints, the final threshold pair is determined according to the bi-objective rule. First, the threshold pair that maximizes the number of problem samples accepted by the system as a whole is selected, i.e., the pair that solves the problem is chosen first.

[0141] .

[0142] If multiple threshold pairs all reach the above maximum value, then among these equally optimal threshold pairs, select the threshold pair that allows the edge-end model to accept the largest number of problem samples, i.e., solve for...

[0143] .

[0144] This yields the final threshold pair. .

[0145] During the testing phase, for any unanswered medical question, the edge model first generates an answer and calculates the corresponding uncertainty score using the aforementioned prediction entropy method; if the uncertainty score of the edge model does not exceed the final first threshold... The system directly accepts the answer from the edge model; if the uncertainty score of the edge model is higher than the final first threshold... If the question is not answered correctly, it will be routed to the cloud model. The cloud model will then generate an answer and calculate the corresponding uncertainty score using the same uncertainty quantification method. If the cloud model's uncertainty score does not exceed the final second threshold... If the cloud model's uncertainty score is still higher than the final second threshold, then the system accepts the answer. If the system fails to output the answer, it will submit the question to the expert review module for manual answering or review.

[0146] Using the above implementation steps, under risk levels of 0.05, 0.10, 0.15, and 0.20, the method of the present invention was applied to the test dataset, and the overall system error detection rates during the testing phase were 0.0436, 0.0925, 0.1412, and 0.1911, respectively. None of these rates exceeded the corresponding preset risk levels, indicating that the method of the present invention achieved overall system error detection rate control on the test set.

[0147] Furthermore, under the aforementioned different risk levels, the proportions of edge models and cloud models accepting the problem are as follows:

[0148] When the risk level is 0.05, the acceptance rate for edge models is 25.65%, and the acceptance rate for cloud models is 26.44%.

[0149] When the risk level is 0.10, the acceptance rate for edge models is 45.39%, and the acceptance rate for cloud models is 21.58%.

[0150] When the risk level is 0.15, the acceptance rate for edge models is 50.60%, and the acceptance rate for cloud models is 33.65%.

[0151] When the risk level is 0.20, the acceptance rate for edge models is 50.87%, and the acceptance rate for cloud models is 46.65%.

[0152] Since the remaining issues not automatically accepted by the system are handled by the expert review module, the corresponding expert review ratios are as follows:

[0153] When the risk level is 0.05, the expert review rate is 47.91%.

[0154] When the risk level is 0.10, the expert review rate is 33.03%.

[0155] When the risk level is 0.15, the expert review rate is 15.75%.

[0156] When the risk level is 0.20, the expert review rate is 2.48%.

[0157] The results of the above implementation demonstrate that the cloud-edge collaborative answering and risk control method for medical question answering proposed in this invention can effectively control the overall error detection rate of the system under different preset risk levels. Furthermore, as the allowable risk level increases, the proportion of questions accepted by the system as a whole increases accordingly. Specifically, the edge model undertakes a significant proportion of question answering tasks with relatively low invocation costs, while for questions with higher uncertainty in the edge model, further processing is performed by the cloud model, thus achieving a balance between system reliability, automated processing capabilities, and invocation costs.

[0158] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0159] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A risk control method for reliable answering of medical knowledge in cloud-edge model collaboration, characterized in that, Includes the following steps: S1: Obtain a medical question and answer sample set, and divide the sample set into a calibration dataset for statistical risk upper bound estimation and threshold calibration and a test dataset for deployment phase verification. Each question sample in the sample set is associated with at least one of the following: reference answer, standard answer, expert answer, or annotation information used to judge the correctness of the answer. S2: For any input medical question, the edge model first generates a corresponding answer and uses a preset uncertainty quantification method to obtain the uncertainty score of the edge model. If the score does not exceed the first threshold, the edge answer is accepted directly; otherwise, it is routed to the cloud model. The cloud model generates a corresponding answer and uses the same uncertainty quantification method to obtain the cloud uncertainty score. If the score does not exceed the second threshold, the cloud answer is accepted; otherwise, the question is submitted to the expert review module. S3: Construct model-level selection indicator variables and model-level error indicator variables to represent whether the question is accepted by the corresponding model and whether the answer generated by the corresponding model is wrong, respectively; S4: Based on the model-level indicator variables, construct system-level selection indicator variables and system-level error indicator variables to represent whether the problem is accepted by the entire cloud-edge collaborative system and whether an error occurs when the problem is accepted by the system, respectively. S5: Given the first threshold and the second threshold, based on the system-level selection indicator variable and the system-level error indicator variable of each problem sample in the calibration dataset, count the number of problems accepted by the system and the number of problems accepted by the system but erroneous, and calculate the empirical statistic of the overall error detection rate of the system. S6: Based on the empirical error detection rate of the calibration dataset, construct an upper confidence bound for the overall error detection rate of the system, wherein the upper confidence bound meets a preset confidence probability requirement; S7: Based on the target risk level set by the user, calculate the upper confidence bound of the overall system error detection rate for each of the multiple candidate first and second threshold combinations, and select a set of candidate threshold pairs whose upper confidence bound does not exceed the target risk level. S8: Among the candidate threshold pairs that meet the risk constraints, the final threshold pair is determined according to the following priority order: the first optimization objective is to maximize the number of problem samples accepted by the system as a whole, and the second optimization objective is to maximize the number of problem samples accepted by the edge model when the number of samples accepted by the system as a whole is the same. S9: Medical questions are processed sequentially through edge models and cloud models, and the final threshold is used to determine whether to accept the corresponding model's answer. Questions that are not accepted are submitted to the expert review module.

2. The method according to claim 1, characterized in that, In step S1, the calibration dataset and the test dataset satisfy the exchangeability condition or the independent and identically distributed condition.

3. The method according to claim 1, characterized in that, In step S2, the uncertainty component method is used to output the corresponding uncertainty score based on the model's response to the input medical question. The larger the uncertainty score, the more uncertain the response is, and the smaller the score, the more certain the response is.

4. The method according to claim 3, characterized in that, The uncertainty quantification method uses the prediction entropy calculated from the category vector obtained based on the logit information of the last layer of the large language model. For a multiple-choice medical question-answering task, the uncertainty score is defined as: ， in, Indicates the number of candidate options. Represented as model for the th The first sample The predicted probability of each option.

5. The method according to claim 1, characterized in that, In step S3: The selection indicator variable for the edge model is defined as: ， in, For the first Marginal uncertainty score of each sample The first threshold, Indicates an indicator function; The error indicator variable for the edge model is defined as follows: ， in, For the edge model, the first The answers of one sample For the first The standard answer for each sample This indicates that the answer is inconsistent with the standard answer; The selection indicator variable for the cloud model is defined as: ， in, For the first Cloud uncertainty score for each sample The second threshold; The error indicator variable for the cloud model is defined as: ， in, For the edge model, the first The responses of one sample.

6. The method according to claim 1, characterized in that, In step S4: The system-level selection indicator variable is obtained by combining the edge model selection indicator variable and the cloud model selection indicator variable, and is defined as follows: ， The system-level error indicator variable is obtained by combining the model selection indicator variable and the corresponding model's prediction error indicator variable, and is defined as follows: 。 7. The method according to claim 1, characterized in that, In step S5, the overall system error detection rate is defined as the ratio of the number of problems accepted by the system that resulted in errors to the total number of problems accepted by the system: ， in The total number of samples in the calibration dataset.

8. The method according to claim 1, characterized in that, In step S6, the upper confidence bound of the overall system error detection rate is denoted as... The upper confidence bound satisfies: ， in, Given a first threshold Second threshold The overall error detection rate of the system below. This is a confidence parameter.

9. The method according to claim 7, characterized in that, In step S6, the Clopper-Pearson method is used to construct an upper confidence bound for the overall error detection rate of the system. This bound is determined when the system accepts a certain number of problems. At that time, the upper confidence boundary is: ， in Let be the quantile function of the Beta distribution. The number of problems accepted by the system and that resulted in errors; when When the threshold is reached, the corresponding threshold pair is considered an infeasible solution.

10. The method according to claim 1, characterized in that, In step S7, the candidate first threshold set consists of the uncertainty scores of all edge models in the calibration dataset, and the candidate second threshold set consists of the uncertainty scores of all cloud models in the calibration dataset.

Images