Prescription comment human-computer collaborative hierarchical routing method and system based on multi-model consensus divergence

Abstract

The application discloses a kind of prescription comment human-computer cooperation hierarchical routing method and system.Patients past medical treatment context is assembled according to preset time window and de-identified, and is issued to N independent large language model according to uniform prompt word and parameter in parallel, each model returns conclusion layer determination result and coding layer problem classification coding set;Conclusions are mapped into ordinal number according to score table, and range is normalized to obtain conclusion layer divergence, coding set is calculated two by two to obtain coding layer divergence, and arithmetic average is obtained by set distance;According to the index and the consistent state of conclusion, the prescription is routed to the artificial review path with different intensity, wherein all models are consistent and reasonable, and are archived and randomly inspected according to fixed proportion, inconsistent and index higher than threshold value are forced to review by senior personnel, and the final conclusion is made by review personnel.The application allocates limited artificial review resources according to determination difficulty, and controls the blind area of the mechanism itself.

Description

Technical Field

[0001] This invention relates to the field of medical information processing technology, specifically to a human-computer collaborative hierarchical routing method and system for prescription review. Background Technology

[0002] Prescription review refers to the retrospective evaluation of prescriptions issued after drug dispensing, determining the rationality of medication use, and categorizing and recording any problems found. This work is performed by pharmacists, and their conclusions carry professional responsibility. In specific medical fields such as psychiatry, prescription review faces particularly significant challenges. On the one hand, the total number of prescriptions is enormous, with a single medical institution issuing tens of thousands of outpatient prescriptions per month. On the other hand, medication use in this field involves numerous off-label uses, combination therapies, and individualized dosage adjustments. The judgment of rationality heavily relies on the patient's past medical history and cannot be based solely on the surface information of a single prescription. Due to manpower limitations, current practices typically only allow for a small, randomly selected percentage of prescriptions for manual review, often less than one percent. The problem with this approach is that the selection is completely random and unrelated to whether the prescription itself might have problems. Consequently, many problematic prescriptions are not selected, while a large number of clearly problem-free prescriptions consume valuable human time. With the development of large language model technology, there have been attempts to use it to assist in judging the rationality of prescriptions. Such approaches typically feed prescription data along with judgment rules into a single model, which then returns a conclusion on whether the prescription is reasonable, and this is then reviewed by humans. However, this approach has a structural flaw: the model's conclusions do not include any information that can be used to judge the reliability of the conclusion itself. When faced with the model's output, humans cannot distinguish which conclusions are credible and which are merely forced responses by the model due to insufficient information or a difficult task. As a result, humans either have to review all model outputs one by one, which fails to alleviate the model's burden, or they only randomly select a portion for review, which is essentially no different from not using a model at all. See also Figure 1 The existing prescription review process is as follows: prescription data 101 is sent to a single large language model for judgment 102, and the model gives a judgment conclusion 103, which is directly presented to the pharmacist 104. Since the conclusion does not include any information that can be used to judge its own reliability, the pharmacist can only review all the conclusions one by one, or randomly sample according to a fixed proportion 105. Neither of these methods can match the difficulty of judging the prescription itself with the human input. To address the reliability measurement problem, several approaches have emerged in existing technologies. One approach utilizes the probability information output by the model itself, such as taking the entropy or variance of the predicted distribution as the confidence level. However, this method is difficult to apply in large language model scenarios: most large language model services do not disclose word probabilities to the caller, and even if they do, there is no stable correspondence between word-level probabilities and whether the overall judgment is correct; it is very common for the model to give incorrect conclusions with high confidence. Another approach is to have the same model sample the same input multiple times, using the dispersion of the sampling results as a measure of uncertainty; this method measures the randomness within the same model and cannot reflect the systematic differences in knowledge coverage and judgment tendencies between different models. Another approach is to introduce multiple models and compare the consistency of their outputs. This type of method has applications in fields such as data cleaning, image recognition, and remote sensing interpretation. The common practice is: when the outputs of multiple models are inconsistent, the sample is marked and submitted for manual processing; when the outputs of multiple models are consistent, the result is considered reliable and is directly adopted or automatically determined. While the above approach utilizes the divergence between models, it still has three shortcomings: First, the methods for measuring consistency are too simplistic. Existing practices often employ a binary judgment of "whether it is consistent" or a single-dimensional measure such as category consistency rate or difference count. However, the output of prescription review models is not a single scalar; it contains at least two completely different levels: one is the conclusion of whether the overall system is reasonable, which is determined by a set of enumerated values ​​ordered by severity, and the discrepancy has direction and magnitude; the other is the set of classification codes for specific problems, and the discrepancy is the difference between the members of the set, without order. The discrepancy between these two levels cannot be measured in the same way, nor can they be simply added together. Existing technology does not provide a solution to synthesize the two into a single comparable scalar. Secondly, the current approach to human intervention is on / off rather than tiered. Existing practices only differentiate between "sending human intervention" and "not sending human intervention," without considering the intensity of intervention or the qualifications of the personnel responsible for review. In real-world scenarios with limited human resources, the real challenge lies in allocating varying levels of human intervention among numerous pending cases, rather than simply deciding whether to intervene at all. Thirdly, and most importantly, current technology invariably adopts or automatically approves any consensus among multiple models. This approach implicitly assumes an untested truth: that consensus among models equates to correct conclusions. However, multiple models may share similar training data and biases, leading to common errors on the same type of problem. These errors do not generate any disagreement and therefore will never come to the attention of human reviewers under the aforementioned mechanism, nor can they be detected through verification of divergent samples. As a result, the hierarchical mechanism itself continuously accumulates an imperceptible systemic blind spot, which expands with the system's runtime. Furthermore, in scenarios involving the invocation of external large language model services, there is an issue that existing technologies have not addressed: the service provider may change its underlying model while maintaining the same interface name. This change is invisible to the caller, but it can cause the judgment criteria to shift without the user's awareness, resulting in incomparable evaluation results between different periods. In summary, current technologies lack a technical solution that can perform divergence quantification on the two-layer outputs of multiple large language models, thereby hierarchically scheduling limited human review resources, while controlling the blind spots of the hierarchical mechanism itself. Summary of the Invention

[0003] 3.1 Technical problems to be solved The technical problem to be solved by this invention is: in the context of prescription review, how to allocate limited human review resources among a large number of prescriptions to be reviewed in a differentiated manner, so that prescriptions that require human input are given priority and receive human input at a higher intensity, while avoiding the generation of imperceptible systemic blind spots in the allocation mechanism itself. 3.2 Technical Solution To solve the above-mentioned technical problems, the present invention adopts the following technical solution. This invention provides a human-computer collaborative hierarchical routing method for prescription review, comprising the following steps. Step 1, Data Access. Obtain the prescription data of the prescription to be reviewed. The prescription data includes at least the patient's basic demographic information, the diagnosis information of this visit, and the name, dosage form, specifications, usage, dosage and quantity of the drugs included in this prescription. Step two, context assembly and de-identification. The patient's past medical history is assembled according to a preset time window to obtain the review object data; then, the review object data is de-identified to remove patient and medical staff identification information. The preset time window uses the business date of the prescription as the anchor point, that is, it traces backward from the date the prescription was issued, rather than from the date the data was processed. This ensures that the same prescription receives the same context at any time it is processed, guaranteeing reproducible results. The time window is a left-closed, right-open interval, meaning it includes the start date of the interval but not the date the prescription was issued, thus avoiding mixing other medical records from the same day into the historical context. In one embodiment, the time window is sixty days. The de-identification process removes, but is not limited to, the following: patient name, patient identification number, outpatient number, medical record number, prescription number, ID card number, contact information, and detailed address. The de-identification process also removes medical personnel identification information, including the name and employee number of the prescribing physician. The purpose of removing medical personnel identification information is to prevent the model's judgment from being influenced by a specific physician's identity, ensuring that the judgment is based solely on the prescription content itself. Step 3: Unified prompt word construction and parallel distribution. A unified prompt word template and unified calling parameters are constructed. Based on this template and parameters, the de-identified comment data is distributed in parallel to N independent large language models, where N is an integer not less than 3. The term "mutually independent" means that the N models are provided by different providers or based on different training corpora and architectures, rather than multiple calls to the same model or multiple randomized copies of the same model. The term "uniformity" is a key limitation of this invention. The prompts, context, output format, and calling parameters received by the N models are completely identical. Its technical significance lies in forming a controlled control: since all input variables are fixed, the observed output differences can be uniquely attributed to the differences in judgment within the models themselves, rather than differences in prompting methods or parameter settings. If the prompts or parameters received by each model are different, then their output differences will be mixed with inseparable interference components, and the degree of divergence cannot be used as a reliable signal of the difficulty of judgment. In one implementation, the calling parameters include: a sampling temperature of 0.1, a maximum number of terms in a single response of 512, and a retry count of 2 when a single model call fails. Using a lower sampling temperature aims to suppress random fluctuations within the same model, ensuring that observed differences originate primarily from differences between models rather than within the model itself. The prompt word template includes a question classification code dictionary. This dictionary is generated at runtime by reading from the data table and concatenating the results, rather than being hard-coded into the prompt word text. Its purpose is to ensure that when the classification criteria are adjusted according to regulations or internal rules, only the data table needs to be updated to synchronize the judgment criteria of all models, without modifying the program code. This also avoids inconsistencies caused by different models using different versions of the dictionary. In one implementation, the question classification codes consist of twenty-eight codes, belonging to three main categories. Step 4, Two-layer structured output parsing. Parse the structured output returned by each model. The structured output includes two layers: The conclusion layer determines whether the overall prescription is reasonable, and its value is taken from a set of enumerated values ​​in order of severity. In one embodiment, there are four enumerated values, ranked from lowest to highest severity: reasonable, non-standard, inappropriate, and abnormal. The order of severity is determined by a pre-defined scoring table, which explicitly maps each enumeration value to an integer ordinal, rather than relying on the order in which the enumeration values ​​are written in the definition. The advantage of using an explicit scoring table is that the severity relationship of the judgment criteria is fixed as a verifiable, verifiable, and individually modifiable configuration that can be adjusted according to regulations, and does not change due to changes in the writing order of the enumeration definitions in the program. The coding layer is a set of classification codes for the problems existing in the prescription. This set can be empty, indicating that the model believes there are no categorizable problems; or it can contain multiple codes, indicating that the model believes there are multiple types of problems simultaneously. The reason for using a dual-level output is that the conclusion and the problem type are two orthogonal levels. Two models may simultaneously determine that a prescription is unreasonable, but point to completely different problems; or they may point to the same problem, but give different conclusions about its severity. Taking only one approach cannot fully characterize the discrepancies between the models. Step 5: Calculation and weighted synthesis of the two-layer divergence degree. Calculate the divergence degree of the conclusion layer and the coding layer separately, and then synthesize them into a comprehensive divergence index using weighted summaries. Calculation of the conclusion layer divergence: Based on the scoring table, the rationality judgment results of each model are mapped to ordinal numbers. Assuming there are L enumerated values, the ordinal numbers range from 0 to L minus 1. The difference between the maximum and minimum values ​​among the N ordinal numbers is taken as the ordinal range, and then normalized by dividing by the maximum possible ordinal difference to obtain the conclusion layer divergence D1, which ranges from 0 to 1. In one implementation, L is set to 4, and the four enumerated values ​​are assigned 0, 1, 2, and 3 respectively for being reasonable, non-standard, unsuitable, and abnormal. The maximum possible ordinal difference is 3, so the normalization is divided by 3. When the number of models involved in the calculation is less than 2, the conclusion layer divergence is set to 0. Using the range instead of variance or standard deviation has the advantage that the range reflects the maximum span of disagreement. In safety-related decision-making scenarios, even if most models agree, if one model concludes that the severity is significantly higher, the prescription should receive greater human attention. Variance is diluted by the consensus of most models, while the range is not. Calculation of coding layer divergence: For the problem classification coding sets of N models, calculate the set distance for each pair, and obtain N multiplied by N minus 1 and then divided by 2 distance values. Take the arithmetic mean of these distance values ​​to obtain the coding layer divergence D2, which ranges from 0 to 1. In one implementation, the set distance is the Jaccard distance, which is the ratio of the number of elements in the symmetric difference between two sets to the number of elements in their union. The distance is 0 when the two sets are completely identical, and 1 when they are completely disjoint. When both sets are empty, their distance is assumed to be 0. The reason for using set distance instead of element-by-element comparison is that there is no order relationship between the elements of the encoded sets, and the number of codes given by different models may vary. Weighted synthesis: The overall divergence index D is obtained by multiplying W1 by D1 and W2 by D2, where W1 and W2 are both greater than zero and their sum is 1. Therefore, the value range of D is still 0 to 1. In one implementation, W1 is set to 0.6 and W2 is set to 0.4, meaning that the weight of the conclusion layer is higher than that of the coding layer. The reason for this value is that the divergence at the conclusion layer is directly related to whether the prescription will be judged as problematic, and its consequences outweigh the differences in problem type classification. Step Six: Parallel Output of Consensus Encoding, Dispute Encoding, and Distribution Pattern. While calculating the divergence index, the following three items are output in parallel: Consensus coding is the intersection of N model problem classification code sets, representing the problem that all models agree upon. The dispute code, which is the difference between the union and intersection of the above sets, represents a problem that is only partially pointed out by the model. The distribution pattern determination result is a discrete label determined based on the number of successfully returned models and their conclusion distribution structure. In one embodiment, the distribution pattern is divided into six categories: Consistent and reasonable, meaning that all successfully returned model judgment results are consistent and reasonable, and the problem classification codes given by all models are empty sets; The consistency is problematic, meaning that all successfully returned model judgments are consistent, but the consistent conclusion is not reasonable, or although it is reasonable, at least one model gives a problem classification code; Mild controversy, meaning that at least three models successfully returned results, and their conclusions were not completely consistent but a majority opinion existed; that is, the number of different conclusions was less than the number of successful models. Significant controversy occurs when three or more models successfully return results with differing conclusions, or when exactly two models successfully return results with differing conclusions, resulting in a tie. Single model only, meaning only one model successfully returned a result; All failed, meaning no model returned successfully. The "consistently reasonable" category includes the condition "the entire encoding set is empty." This is to prevent misclassification as "no problem" when a model provides a reasonable conclusion while simultaneously pointing out a specific problem. Such contradictory outputs inherently indicate uncertainty in the prescription and should not be classified as "no problem." The technical advantage of consensus coding and dispute coding lies in locating disagreements from a scalar quantity to specific issue items. During manual review, it's possible to directly see which type of issue the disagreement occurred on, without needing to manually compare the complete outputs of each model. In one implementation, the manual judgment workbench distinguishes dispute codes, for example, by using different colors or adding annotations. The reason why the distribution pattern and the divergence index are generated in parallel rather than derived from the divergence index is that they carry different information: the divergence index is a continuous quantity, reflecting the magnitude of the divergence; the distribution pattern is a discrete quantity, reflecting the structure of the divergence. The same divergence index may come from completely different distribution structures. For example, "one model strongly opposes the other models" and "all models hold opposing views" may yield similar index values, but the manual processing required for the two is different. Step 7, Hierarchical Routing. Based on the consistency between the comprehensive divergence index and the judgment results of each model, the prescription is routed to review paths with different levels of human intervention. The routing includes the following branches: Branch 1: If not all N models successfully return structured output, the output of this model is not accepted, and the prescription will undergo regular manual review. The reason for setting this branch is that when some models fail to return results due to network issues, rate limiting, or service anomalies, the divergence index calculated based on the remaining models is not comparable and cannot be used as a routing basis. Branch Two: If all models' judgments are consistent and all are reasonable, the prescription is archived, and a portion of these archived prescriptions is randomly selected for manual review according to a preset ratio. In one implementation, the preset ratio is 5%. Furthermore, this branch requires that all models provide an empty set of problem classification codes. If any model provides a problem classification code while giving a reasonable conclusion, the prescription is not included in this branch but is processed according to Branch Three. Branch 3: If all models consistently determine the prescription as unreasonable, then the prescription will undergo routine manual review. Although there is no disagreement with such prescriptions, since all models point to a problem, manual verification is required to confirm whether the problem is valid and its severity. Branch 4: If the judgment results of each model are inconsistent and the overall disagreement index is not higher than the preset threshold, the prescription will enter the regular manual review and the dispute code will be displayed differently in the workbench. Branch 5: If the judgment results of each model are inconsistent and the overall discrepancy index is higher than the preset threshold, the prescription will enter the mandatory senior personnel review path. That is, the prescription cannot be closed by general reviewers alone, but must be reviewed by personnel with corresponding qualifications. In one implementation, the preset threshold is set to 0.5. The random sampling in Branch Two is a key feature of this invention, its technical effect being to control the blind spots of the grading mechanism itself. If all identical prescriptions are directly archived, misjudgments caused by the shared bias of multiple models will not generate any divergence signals and will never be detected manually. Such misjudgments accumulate undetectably during system operation. With a fixed proportion of random sampling, such misjudgments still have a way to be detected: once a completely identical but actually unreasonable prescription is found during sampling, it can be determined that the grading mechanism has failed on that type of prescription, and the prompt words, coding dictionary, or threshold can be adjusted accordingly. In other words, this setting makes the grading mechanism itself an object that can be continuously tested, rather than an unobservable black box. In each branch, the model's structured output is presented only as reference information. The final review conclusion for the prescription is made by the reviewer, and the reviewer's identification is recorded. The system does not directly generate publicly released review conclusions based on the model output. Step 8, Audit Binding. Calculate a hash value by combining the prompt word template, the calling parameters, and the model version identifier actually returned by each model in its response. Then bind and store this hash value with the structured output and comprehensive divergence index of this operation. The key here is that the model version identifier used in the hash calculation is taken from the field actually returned in the response message of each model, rather than from the interface name filled in the local configuration file. The technical effect is that the provider of the large language model service can change its underlying model while keeping the interface name unchanged; this change is invisible to the caller. If the hash is only bound to the local configuration, the hash value remains unchanged after the underlying model is changed, and the judgment criterion has drifted without the system's detection. By binding it to the actual returned version identifier, once such a change occurs, the hash value changes immediately, and the records from the two periods can be distinguished, thus allowing the drift of the judgment criterion to be detected and traced. Step Nine: Controlling the Presentation Order During Independent Review by Multiple Personnel. When the same prescription is independently reviewed by two or more personnel, the presentation order of each model's output is deterministically shuffled according to the reviewer's identification. "Deterministic" means that the same reviewer sees the same order each time they review the same prescription, while different reviewers see different orders. The technical effect is to avoid a fixed presentation order that could lead to a stable bias among reviewers towards models listed first, while ensuring consistency in what the same person sees when reviewing repeatedly, thus preventing confusion. Step 10, Tie Resolution. When a tie occurs due to majority voting based on the results of each model, the vote with the higher severity is taken. This setting allows the system to lean towards the more cautious side when a majority opinion cannot be formed. 3.3 Beneficial Effects The beneficial effects of this invention are as follows. First, it ensures that the allocation of manual review resources is based on the difficulty of judging the prescription itself rather than random sampling. The divergence index is a continuous scalar and is normalized, so different prescriptions can be compared horizontally and a priority order can be formed, which cannot be provided by binary judgment of "whether they are consistent" and unnormalized difference count. Second, a two-layer heterogeneous measure with weighted synthesis is adopted to include both the disagreement at the conclusion level and the disagreement at the problem type level, with their relative importance reflected by pre-set weights. Compared to a single-dimensional consistency measure, this scheme provides a more complete characterization of disagreement. Third, disagreements are not resolved but are retained and used for scheduling. Unlike the approach of merging the outputs of multiple models into a single "more credible answer", this invention does not modify the conclusions of any model. Instead, it uses the disagreements themselves as scheduling signals, and locates the disagreements to specific items through dispute coding, so that humans can directly work on the points of contention. Fourth, the blind spots of the grading mechanism itself are controlled. By retaining a fixed proportion of random sampling of prescriptions that are unanimously and reasonably judged, misjudgments caused by biases shared by multiple models still have a channel for detection, thus making the grading mechanism a mechanism that can be continuously tested. Fifth, it enables the detection of shifts in decision criteria. By incorporating the actual version identifier returned by each model into hash binding, it is possible to identify situations where the service provider changes the underlying model while keeping the interface name unchanged, thus distinguishing records from different periods. Sixth, the authority to make judgments always rests with humans. The system does not automatically generate publicly released review conclusions under any branch; the model output is presented only as reference information, and the final conclusion is made by a named reviewer. This ensures that this approach meets the professional responsibility requirements for prescription review. The aforementioned beneficial effects are based on the technical rationality of the mechanism and operational observations in a real data environment. In one implementation environment, approximately 168,000 real outpatient prescriptions have been accessed and structured, along with approximately 600,000 previous medical history records, resulting in a prescription pool of 250,609 prescriptions available for analysis, covering nine months with an average of approximately 27,845 prescriptions per month. Fifty-two complete tri-model consensus records have been completed in this environment, with their overall divergence index values ​​ranging from 0.000 to 0.800. Thirty-nine records (75.0%) have non-zero divergence indices. Based on distribution patterns, thirty-one records show slight divergence, twelve are all consistent and reasonable, seven are all consistent and unreasonable, and two show severe divergence. The version hashes of these fifty-two records are completely identical, indicating that the judgment conditions did not drift during this period. Regarding the de-identification effect, three hundred real prescriptions have been verified one by one, and the number of times patient identification number, outpatient number, medical history number, and prescription number were entered into the model's content is zero. The above data shows that the divergence index exhibits sufficient discriminative power in real data rather than tending to a constant, thus possessing practical feasibility for hierarchical scheduling. Attached Figure Description

[0004] Figure 1 is a schematic diagram of the prescription review process in the prior art; Figure 2 is a block diagram of the system of the present invention; Figure 3 is a flowchart illustrating the calculation of the comprehensive divergence index in this invention; Figure 4 is a flowchart of the hierarchical routing decision process in this invention. Detailed Implementation

[0005] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. The following examples are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Example 1: System Composition Referring to Figure 2, the prescription review human-machine collaborative hierarchical routing system provided in this embodiment includes: Data access module 1 establishes a read-only connection with the hospital information system to acquire prescription data for prescriptions to be reviewed on a daily or batch basis; Context assembly module 2 assembles the patient's past medical history according to a preset time window and performs de-identification processing; The prompt word construction module 3 reads the question classification code dictionary from the data table and concatenates it to generate a unified prompt word template, while also determining unified calling parameters; Model 41, Model 42, and so on up to Model 4N are N independent large language models that receive the same prompt words and parameters and return structured output independently. Output parsing module 5 parses the structured output returned by each model, separating the conclusion layer judgment results from the coding layer problem classification code set; models that fail to parse or fail to return within the timeout period are marked as failures. The divergence calculation module 6 executes the process shown in Figure 3, producing a comprehensive divergence index, consensus code, dispute code, and distribution pattern determination results. The hierarchical routing module 7 executes the process shown in Figure 4, assigning prescriptions to different manual review paths; The manual judgment workbench 8 presents the prescription content and model output as reference information to the reviewers, and receives and records the final comments and labels made by the reviewers; The audit binding module 9 calculates a hash by combining the prompt word template, calling parameters, and the version identifier actually returned by each model, and binds and stores this hash with the current result. Example 2: Method Flow Referring to Figure 3, the calculation process for the comprehensive divergence index is as follows. S1, obtains the structured outputs of N models. S2, Conclusion Layer Processing: Based on the scoring table, the rationality judgment results of each model are mapped to ordinal numbers. Taking the four-level enumeration as an example, rationality is mapped to 0, non-standardity to 1, inappropriateness to 2, and abnormality to 3. S3, take the difference between the maximum and minimum values ​​of the N ordinal numbers, divide it by the largest possible ordinal difference (3 in this example) and normalize it. S4 yields the conclusion level divergence degree D1. S5, Encoding layer processing: Calculate the Jaccard distance pairwise for each of the N problem classification encoding sets. S6, take the arithmetic mean of the distances obtained from all pairwise combinations. S7, obtain the coding layer divergence degree D2. S8, the weighted composite divergence index is calculated by multiplying D by W1 and D1 and then by multiplying W2 by D2. S9, in parallel with the above, determines the distribution pattern of the output based on the determination results of each model. S10 outputs the comprehensive divergence index and distribution pattern to the hierarchical routing module. S11, in parallel with the above, produces consensus codes and dispute codes. Referring to Figure 4, the decision-making process for hierarchical routing is as follows. T1 yields the comprehensive divergence index. T2 determines whether all N models have returned successfully. T3. If not, it will not be accepted and will proceed to regular manual review. T4, if yes, determine whether the conclusions of each model are all consistent. T5. If all are consistent, further determine whether the consistent conclusion is reasonable. T6. If the consensus is that the conclusion is reasonable, the document will be archived and randomly sampled at a fixed ratio. T7. If the consensus is that the conclusion is unreasonable, proceed to routine manual review. T8. If the conclusions are inconsistent, determine whether the overall disagreement index is higher than the set threshold. If T9 is not higher than the threshold, it will proceed to regular manual review and the disputed codes will be displayed separately. If T10 is higher than the threshold, the system will proceed to the mandatory senior staff review process. Example 3: Numerical Example Let N be 3, L be 4, W1 be 0.6, W2 be 0.4, the threshold be 0.5, and the sampling rate be 5%. Scenario 1: The conclusions of the three models are reasonable, reasonable, and reasonable, respectively, with ordinal numbers of 0, ranges of 0, and D1 equal to 0; the problem encoding sets of all three are empty sets, the pairwise Jaccard distances are all 0, D2 equals 0, and D equals 0. The prescription conclusions are consistent, all reasonable, and all encoding sets are empty. The distribution pattern is determined to be consistent and reasonable. It is archived according to branch two and has a 5% probability of being randomly selected for manual review. Scenario 2: The conclusions of the three models are reasonable, non-standard, and non-standard, respectively, with ordinal numbers of 0, 1, and 1, a range of 1, and D1 equals 1 divided by 3, approximately 0.333; the coding sets are empty, {A01}, and {A01}, respectively, with pairwise Jaccard distances of 1, 1, and 0, an average of 0.667, and D2 equals 0.667; D equals 0.6 multiplied by 0.333 plus 0.4 multiplied by 0.667, approximately 0.467. This value is not higher than the threshold of 0.5, the distribution pattern is judged as slightly controversial, and it enters the routine manual review according to branch four. The controversial code A01 is displayed differently in the workbench. It should be noted that although two models in this case both provide code A01, the consensus code is the strict intersection of the codes of all successful models. The third model provides an empty set, therefore the consensus code is an empty set, and A01 is classified as a disputed code. The purpose of using a strict intersection rather than a majority intersection is to ensure that the issue is presented as a consensus only when all models point to the same problem, thus avoiding prematurely solidifying the majority opinion as a conclusion. Any item with a differing opinion is subject to manual judgment. Scenario 3: The conclusions of the three models are reasonable, non-standard, and abnormal, respectively, with ordinal numbers of 0, 1, and 3, a range of 3, and D1 equals 1; the encoding sets are empty, {A01}, and {B01,C01}, respectively; the pairwise Jaccard distances for all three pairs are 1, the average is 1, D2 equals 1, and D equals 1. This value exceeds the threshold, and the distribution pattern is judged as highly controversial, thus entering the mandatory senior personnel review path according to branch five. Example 4: Deformation Method The technical solution of the present invention can have various modifications, including but not limited to the following. In variation one, the number of models N can be any integer, either 2 or greater than 3. When N is 2, there is only one pairwise combination in the coding layer, and the arithmetic mean is the distance between these pairs. The larger the value of N, the better the stability of the divergence, but the calling cost increases accordingly. Variation 2: The division of the conclusion layer enumeration values ​​can use different levels and names, as long as the order of severity is maintained. The normalization divisor is taken as the level minus 1. Variation 3: The distance between the coding layers can be measured using other metrics besides the Jaccard distance, such as the complement of the Dice coefficients, the complement of the overlap coefficients, or by converting the coding set into an indicator vector and then calculating and normalizing the Hamming distance. Variation 4: The values ​​of W1 and W2 can be adjusted within a range greater than zero and summing to 1. In scenarios that place greater emphasis on problem type identification, W2 can be increased. Variation 5: The coding layer divergence can also be calculated by taking the maximum value of all pairwise distances instead of the arithmetic mean, so that significant deviations of individual models can be given higher weights. Variation 6: The threshold can be set to two or more levels, corresponding to more levels of human intervention intensity. For example, those below the first threshold are reviewed by general staff, those between the two thresholds are reviewed by general staff and confirmed a second time, and those above the second threshold are reviewed by senior staff. Variation 7: The proportion of random sampling can be dynamically adjusted based on historical review results. For example, the sampling proportion can be increased when the false positive rate found in the sampling increases and decreased when it decreases. Variation 8: The number and classification methods of distribution patterns can be adjusted according to actual needs. Modification 9: The length of the time window can be adjusted according to the characteristics of the disease, and different time window lengths can be used for different drug categories. Variation 10: Consensus coding can also take the coding given by most models rather than the strict intersection of all models, in order to achieve a different balance between consensus and prudence. In variation eleven, the normalized divisor can also be dynamically determined based on the actual number of entries in the scoring table, that is, the difference between the maximum and minimum ordinal numbers in the scoring table, thereby automatically adapting when adjusting the severity level. Modification 12: In addition to being applicable to the method described in claim 1, the above-mentioned preferred features are also applicable to the system described in claim 2, that is, the corresponding modules of the system can be configured to perform the processing corresponding to the above-mentioned preferred features. The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A human-computer collaborative hierarchical routing method for prescription review, characterized in that... ,include: Obtain the prescription data of the prescriptions to be reviewed, and assemble the patient's past medical history according to a preset time window to obtain the review object data; The data of the review subjects is de-identified, removing patient and medical staff identification information; Based on a unified prompt word template and unified calling parameters, the de-identified comment object data is distributed in parallel to N independent large language models, where N is an integer not less than 3. Each large language model independently returns structured output. The structured output includes the reasonableness judgment result of the conclusion layer and the problem classification code set of the coding layer. The reasonableness judgment result is taken from a set of enumerated values ​​with a severity order. The reasonableness judgment results are mapped to ordinal numbers in order of severity. The difference between the maximum and minimum values ​​among the N ordinal numbers is taken and normalized by dividing by the maximum possible ordinal difference of the enumerated values ​​to obtain the conclusion level divergence degree D1. For each of the N problem classification coding sets, calculate the set distance between each pair and take the arithmetic mean of all pairwise combinations to obtain the coding layer divergence degree D2; The weighted composite divergence index D is calculated as D = W1×D1 + W2×D2, where both W1 and W2 are greater than zero, and the sum of W1 and W2 is 1. Based on the consistency between the comprehensive disagreement index and each of the reasonableness judgment results, the prescription is routed to review paths with different levels of human intervention. The routing includes: If all the reasonableness judgment results are consistent and all are reasonable, the prescription will be archived, and a portion of the archived prescriptions will be randomly selected for manual review according to a preset ratio. If all the reasonableness assessment results are consistent and all are deemed unreasonable, the prescription will proceed to routine manual review. If the reasonableness judgment results are inconsistent and the comprehensive disagreement index is not higher than a preset threshold, the prescription will proceed to routine manual review. If the reasonableness judgment results are inconsistent and the comprehensive disagreement index is higher than the preset threshold, the prescription will enter the mandatory senior personnel review path. In each of the review paths, the structured output is presented only as reference information, and the final review conclusion of the prescription is made by the reviewer, and the reviewer's identification is recorded.

2. A human-computer collaborative hierarchical routing system for prescription review, characterized in that... ,include: The data access module is configured to acquire prescription data for prescriptions to be reviewed. The context assembly module is configured to assemble the patient's past medical history according to a preset time window to obtain the review object data, and to perform de-identification processing on the review object data to remove the patient identification information and medical staff identification information. The prompt word construction module is configured to generate a unified prompt word template and unified calling parameters; The model invocation module is configured to, based on the prompt word template and the invocation parameters, distribute the de-identified review object data in parallel to N independent large language models, where N is an integer not less than 3; The output parsing module is configured to parse the structured outputs independently returned by each of the large language models to obtain the reasonableness judgment results of the conclusion layer and the problem classification code set of the coding layer. The reasonableness judgment results are taken from a set of enumerated values ​​with a severity order. The divergence calculation module is configured to map each of the rationality judgment results into ordinal numbers according to the severity order and take the ordinal range, divide it by the maximum possible ordinal difference of the enumerated values ​​to normalize and obtain the conclusion layer divergence degree D1; calculate the set distance of each pair of the N problem classification coding sets and take the arithmetic mean to obtain the coding layer divergence degree D2; and synthesize a comprehensive divergence index D by weighting according to D = W1×D1 + W2×D2, where W1 and W2 are both greater than zero and their sum is 1; The hierarchical routing module is configured to route the prescription to review paths with different levels of manual intervention based on the consistency between the comprehensive disagreement index and each of the reasonableness judgment results. The routing includes: when all reasonableness judgment results are consistent and all are reasonable, the prescription is archived, and a portion of the archived prescriptions is randomly selected for manual review according to a preset ratio; when all reasonableness judgment results are consistent and all are unreasonable, the prescription proceeds to regular manual review; when the reasonableness judgment results are inconsistent and the comprehensive disagreement index is not higher than a preset threshold, the prescription proceeds to regular manual review; when the reasonableness judgment results are inconsistent and the comprehensive disagreement index is higher than the preset threshold, the prescription proceeds to a mandatory senior personnel review path. The manual judgment workbench is configured to present the structured output as reference information only, receive the final review conclusion of the prescription made by the reviewer, and record the identifier of the reviewer.

3. The method according to claim 1, characterized in that... There are L enumerated values ​​in total, where L is an integer not less than 2, and the maximum possible ordinal difference is L minus 1.

4. The method according to claim 1, characterized in that... The set distance is the Jaccard distance, which is the ratio of the number of elements in the symmetric difference between the two problem classification coding sets to the number of elements in the union of the two sets.

5. The method according to claim 1, characterized in that... It also includes: taking the intersection of the N problem classification code sets as the consensus code, and taking the difference between the intersection and the union of the N sets as the dispute code; the consensus code and the dispute code are generated in parallel with the comprehensive disagreement index, and the dispute code is displayed separately in the manual review.

6. The method according to claim 1, characterized in that... It also includes: producing discrete distribution pattern determination results based on the distribution of each of the rationality determination results, wherein the distribution pattern determination results are produced in parallel with the comprehensive divergence index; and the routing is further performed based on the distribution pattern determination results.

7. The method according to claim 1, characterized in that... It also includes: calculating a hash value by combining the prompt word template, the calling parameters, and the model version identifier actually returned by each of the large language models in its response, and binding and storing the hash value with the current structured output and the comprehensive divergence index.

8. The method according to claim 1, characterized in that... The preset time window uses the business date of the prescription as the anchor point and takes a time interval that is closed on the left and open on the right.

9. The method according to claim 1, characterized in that... The W1 is set to 0.6, the W2 is set to 0.4, the preset threshold is set to 0.5, and the preset ratio is set to five percent.

10. The method according to claim 1, characterized in that... It also includes: if the N large language models fail to return the structured output successfully, the current structured output will not be accepted, and the prescription will be subject to routine manual review.

Images