Intelligent matching method and system for patients in multi-stage recommendation clinical trials

By employing a multi-stage recommendation method and a multi-agent feedback optimization mechanism, combined with neural reordering and a large language model for clinical trial patient matching, the problems of matching errors and performance fluctuations in existing technologies have been solved, achieving an efficient and controllable optimization process.

CN123120366APending Publication Date: 2026-11-17HEFEI UNIV OF TECH +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202611629952.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-21
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies suffer from matching errors and performance fluctuations in patient matching during clinical trials. They are inefficient, rely on manual analysis, and lack reproducibility in the optimization process.

Method used

A multi-stage recommendation approach is adopted, using a neural reordering model for initial matching, combined with a large language model for clinical review, and a multi-agent feedback optimization mechanism for model updating and validation, ensuring that each update has clear diagnostic basis and is controllable.

Benefits of technology

This improves the efficiency of iterative optimization of patient matching models, reduces the computational overhead of repeated debugging, and ensures the stability and reliability of matching performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123120366A_ABST
    Figure CN123120366A_ABST
Patent Text Reader

Abstract

This invention provides a method and system for intelligent matching of patients in multi-stage recommended clinical trials, relating to the fields of medical information processing and artificial intelligence. The invention consists of three main stages: the first stage is a neural network reordering model, used to generate initial matching scores from the patient set; the second stage is a large language model verification module, used to perform clinical semantic verification on the reordered candidate trial-patient pairs and output structured verification results; the third stage is a multi-agent feedback optimization mechanism, used to diagnose, make decisions, update, and verify the neural network reordering model based on the neural network reordering results, the large model verification results, the true labels, and validation metrics. Through this three-stage framework of neural network reordering—large model verification—feedback optimization, the automation, controllability, and auditability of the model optimization process are greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of medical information processing and artificial intelligence technology, specifically to a method and system for intelligent matching of patients in multi-stage recommended clinical trials. Background Technology

[0002] Patient matching in clinical trials typically requires comparing the target disease, inclusion criteria, and exclusion criteria of the clinical trial with multi-source information in the patient's medical records, including diagnosis, chief complaint, present illness, examination results, past medical history, and medication history. Because the aforementioned text contains medical terminology, negative expressions, numerical conditions, and cross-field associations, relying solely on keywords or manual rules can easily lead to insufficient candidate recall or inaccurate eligibility criteria. Conversely, directly calling a large language model on the entire patient population incurs high computational costs and response latency.

[0003] To balance matching accuracy and computational efficiency, related technologies propose a patient screening scheme that utilizes both large and small models in synergy. For example, patent application CN121862284A discloses a method and system for intelligently mining potential clinical trial subjects using a combination of large and small models. This method includes: preprocessing patient data and clinical trial data; generating a bag-of-words clinical query from the large model based on the trial name and abstract; obtaining candidate patients through BM25 retrieval; outputting and ranking the first matching probability and labels of candidate patients using the small model; forming a review interval based on the trial recruitment quota; and only calling the large model to obtain a second matching probability and labels for patients at the boundary of the interval; finally, fusing the results of the large and small models and forming a recommendation list based on the recruitment quota. This scheme reduces the amount of large model calls and improves the semantic judgment ability of boundary samples.

[0004] The aforementioned approach is primarily used for screening, verifying, and outputting results for candidate patients. However, it still relies heavily on manual analysis and repeated debugging to address matching errors and performance fluctuations that occur during model application. This approach is inefficient and highly dependent on the experience of the personnel involved. Furthermore, the effectiveness of model adjustments is somewhat uncertain, making it difficult to guarantee the stability of matching performance, and the reproducibility of the optimization process needs improvement. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a method and system for intelligent matching of patients in multi-stage recommended clinical trials. It solves the problems of improving the error correction efficiency and update reliability of patient matching model iterative optimization in clinical trial patient matching scenarios, reducing the computational overhead of repeated debugging and the risk of matching performance degradation caused by improper updates.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for intelligent matching of patients in multi-stage recommended clinical trials, comprising: Obtain patient-trial samples with real-match tags, including subject texts and eligibility criteria texts for both patients and clinical trials; Two types of topic samples are input into the neural reordering model. The topic relevance is ranked, and topic relevance scores are output; the two types of qualification standard texts are input into the neural re-ranking model. The relevance scores of the eligibility criteria are ranked and output as eligibility criteria relevance scores. The re-ranked scores are then combined with the scores of the first two criteria and ranked to screen samples to be reviewed. Here, k is the model version number. The scores and rankings of the samples to be reviewed are used as auxiliary prior information. They are then input into a large language model along with the corresponding patient-trial samples for clinical review. The output is a structured review result that includes review scores, review labels and clinical dimension review information. Based on the true matching labels of the samples to be reviewed, the scores and rankings of each item, and the structured review results, a feedback record is constructed, and the error type and deviation direction are determined by the feedback diagnostic agent to obtain effective diagnostic results. Based on the deviation distribution of effective diagnostic results, the decision control agent selects an update mode of strict matching update, relaxed matching update, parameter modification or no change, and generates an execution specification that limits the updatable configuration items. The first three update modes are used to add predefined clinical input fields and retrain the model, reduce predefined unnecessary clinical input fields and retrain the model, and adjust the score fusion coefficient and / or matching judgment threshold without retraining. The updated neural reordering model is generated by the executing agent based on the execution specification. And compared by the verifying agent and Based on the corresponding evaluation metrics on the same fixed validation set, acceptance is determined. , roll back to Alternatively, manual verification can be performed, and the selected model can output the patient-trial matching score and predicted matching label.

[0007] Secondly, the present invention provides an intelligent patient matching system based on multi-stage recommended clinical trials, comprising: The sample acquisition module is used to acquire patient-trial samples with real-match labels, including the subject text and eligibility criteria text of each patient and clinical trial. The reordering module is used to input two types of topic samples into the neural reordering model. The topic relevance is ranked, and topic relevance scores are output; the two types of qualification standard texts are input into the neural re-ranking model. The relevance scores of the eligibility criteria are ranked and output as eligibility criteria relevance scores. The re-ranked scores are then combined with the scores of the first two criteria and ranked to screen samples to be reviewed. Here, k is the model version number. The large language model review module is used to take the scores and rankings of the samples to be reviewed as auxiliary prior information, and input them along with the corresponding patient-trial samples into the large language model for clinical review. The output is a structured review result including review score, review label and clinical dimension review information. The feedback record construction and feedback diagnosis module is used to construct feedback records based on the true matching labels of the sample to be reviewed, the scores and rankings of each item, and the structured review results. The feedback diagnosis agent determines the error type and deviation direction to obtain effective diagnostic results. The decision control module is used by the decision control agent to select an update mode of strict matching update, relaxed matching update, parameter modification or no change based on the deviation distribution of effective diagnostic results, and generate an execution specification that limits the updatable configuration items. The first three update modes are used to add predefined clinical input fields and retrain the model, reduce predefined unnecessary clinical input fields and retrain the model, and adjust the score fusion coefficient and / or matching judgment threshold without retraining. The execution update and verification module is used by the execution update agent to generate the updated neural reordering model based on the execution specification. And compared by the verifying agent and Based on the corresponding evaluation metrics on the same fixed validation set, acceptance is determined. , roll back to Alternatively, manual verification can be performed, and the selected model can output the patient-trial matching score and predicted matching label.

[0008] Thirdly, the present invention provides a storage medium storing a computer program, wherein the computer program causes a computer to execute the intelligent patient matching method based on multi-stage recommended clinical trials as described above.

[0009] Fourthly, the present invention provides an electronic device, the electronic device comprising: A processor and a memory; the memory stores program instructions; the processor is configured to run the program instructions to execute the intelligent patient matching method based on multi-stage recommendation clinical trials as described above.

[0010] (III) Beneficial Effects This invention provides a method and system for intelligent patient matching based on multi-stage recommended clinical trials. Compared with existing technologies, it has the following advantages: This invention constructs a three-stage framework of neural reordering—large-scale model review—feedback optimization. First, the neural reordering model outputs the topic relevance score, eligibility criterion relevance score, reordering score, and ranking of patient-trial samples, and filters samples to be reviewed. Second, the scores and rankings of the samples to be reviewed are used as auxiliary priors, and along with the corresponding patient-trial samples, are input into a large language model for clinical review, generating structured review results. Finally, a multi-agent feedback optimization mechanism is invoked. First, a feedback record is constructed based on the true matching labels, scores and rankings of the samples to be reviewed, and the structured review results. The feedback diagnostic agent uses this record to determine the error type and deviation direction to accurately guide the correction and improve the correction efficiency. The decision control agent selects a strict matching update, a relaxed matching update, or parameter modification or no change update mode based on the deviation distribution, and generates an execution specification that limits the updatable configuration items. The execution update agent generates the updated neural reordering model based on the execution specification. The verification agent compares the old and new model metrics on a fixed verification set to decide whether to accept, roll back, or manually verify, avoiding improper updates that could lead to performance degradation. At the same time, it ensures that each model update has clear diagnostic basis, execution record, and verification results, improving the controllability of the optimization process. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating an intelligent patient matching method based on multi-stage recommendation clinical trials, provided in this embodiment of the invention; Figure 2 A block diagram illustrating an intelligent patient matching method based on multi-stage recommended clinical trials, provided as an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0013] Component labeling explanation: 100 - electronic device, 101 - memory, 102 - processor, 103 - display. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] To address the aforementioned technical problems, this invention proposes a patient intelligent matching method based on multi-stage recommendation clinical trials, the overall approach of which is as follows: This method consists of three main stages. The first stage is a neural network reordering model, used to generate initial matching scores from the patient set. The second stage is a large language model verification module, used to perform clinical semantic verification on the neurally reordered candidate trial-patient pairs and output structured verification results. The third stage is a multi-agent feedback optimization mechanism, used to diagnose, make decisions, update, and validate the neural network reordering model based on the neural reordering results, large model verification results, real labels, and validation metrics. The overall process includes three stages: neural reordering, large model verification, and feedback optimization. The overall framework is as follows: Figure 1 As shown.

[0016] More specifically, the key points of this invention include at least the following aspects: 1) Construct a multi-stage patient matching process: neural reordering, large model review, and feedback optimization.

[0017] The overall process of this invention comprises three main steps. First, a neural re-ranking model is used to rank candidate trial-patient pairs to obtain initial matching scores. Second, a large-scale model is introduced to clinically verify high-potential candidate samples. Finally, the verification results and neural re-ranking results are used together for subsequent feedback optimization. The key to this process is to avoid directly calling the large-scale model on the entire set of clinical trial-patient combinations. Instead, it first uses a neural re-ranking model for matching, followed by clinical verification using the large-scale model, thus balancing matching efficiency, semantic accuracy, and clinical interpretability.

[0018] 2) Use the output of the neural reordering model as an auxiliary prior for the large model review.

[0019] This invention improves upon traditional neural re-ranking models, enabling them to output not only the final re-ranking score but also multi-dimensional intermediate results such as ranking position, topic relevance score, and eligibility criterion matching score. These multi-dimensional outputs are converted into natural language and input into the large model's prompts as supplementary reference information during review. Simultaneously, to reduce the anchoring influence of prior information on the large model's judgments, the prompts explicitly require the large model to prioritize independent judgment based on the original clinical trial text and patient medical records, with the neural re-ranking results serving only as supplementary evidence. This design allows the large model to utilize both the initial ranking information provided by the neural model and to conduct independent review based on original clinical evidence.

[0020] 3) Design a structured clinical review output for a large-scale model.

[0021] This invention does not require large models to output simple "match" or "mismatch" conclusions. Instead, it requires them to output structured review results, including the final inclusion probability, whether inclusion is recommended, disease matching degree, inclusion criterion satisfaction, exclusion risk, agreement with the neural reordering model's judgment, and natural language interpretation. Specifically, disease matching degree measures the correlation between the patient's primary diagnosis, relevant medical history, and the target disease; inclusion criterion satisfaction measures whether the patient's clinical information meets the trial inclusion requirements; exclusion risk measures whether the patient is likely to meet the exclusion criteria; and agreement with the neural reordering model's judgment indicates the large model's attitude towards reviewing the results of preceding models. This structured output makes the large model's review results computable, traceable, and interpretable, providing a basis for subsequent diagnostic feedback and updated decisions.

[0022] 4) Combine the neural reordering results and the large model verification results to form the feedback optimization input.

[0023] The feedback optimization of this invention does not rely solely on the output of the neural reordering model itself, nor solely on the judgment of the large model alone, but integrates both into a unified feedback record. Each feedback record corresponds to a clinical trial-patient candidate sample, including basic sample information, true label, neural reordering score, large model review score, large model review label, disease matching degree, inclusion criterion satisfaction, exclusion risk, whether the neural reordering model is agreed upon, and natural language review reasons, etc. By constructing feedback records, the output results of the neural network reordering model and the large model can be systematically integrated, facilitating subsequent analysis by the feedback diagnostic agent.

[0024] 5) Construct a multi-agent feedback optimization mechanism under rule constraints.

[0025] The multi-agent collaborative mechanism in this invention is mainly reflected in the feedback optimization stage. Specifically, this invention sets up multiple agents, such as a feedback diagnosis agent, a decision control agent, an execution update agent, and a verification agent, which collaboratively make decisions around the same batch of feedback records.

[0026] The feedback diagnostic agent is responsible for identifying sample-level error types, such as error inclusion, error exclusion, calibration error, or no obvious error; the decision control agent is responsible for selecting the update mode based on the error distribution and preset rules, such as keeping it unchanged, strict update, lenient update, or calibration update; the execution update agent is responsible for making minimal modifications to the neural reordering model code according to the structured execution specifications; and the verification agent is responsible for comparing the validation set metrics before and after the modification and determining whether the modification result should be accepted, rolled back, or submitted for manual review.

[0027] The key to this mechanism lies in the collaborative division of labor among multiple intelligent agents to form a controllable feedback optimization decision-making process, rather than relying on a single model or human experience to directly determine model modifications. Simultaneously, this invention avoids performance risks caused by unconstrained automatic modifications by strictly limiting the modifiable content and update patterns through rule constraints: for strict and lenient updates, adjustments to the input fields of the neural re-ranking model are only allowed within a predefined range of candidate fields, requiring retraining of the relevant model; for calibration updates, the system only allows adjustments to the fusion coefficient α or classification threshold of the final score of the neural re-ranking model, without altering the model's core structure; for mode preservation, no code modifications are performed.

[0028] Furthermore, executing the update agent requires backing up the original code, outputting the modified results to a new results directory, and saving the execution log, modification prompts, standard output, standard error, and runtime summary. This design avoids arbitrary modifications to the training process, evaluation logic, or data path, improving the reproducibility of the feedback optimization process.

[0029] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0030] like Figure 2 As shown, this embodiment of the invention provides a method for intelligent matching of patients based on multi-stage recommended clinical trials, including: S1. Obtain patient-trial samples with real-match tags, including subject texts and eligibility criteria texts for both patients and clinical trials; S2. Input the two types of topic samples into the neural reordering model. The topic relevance is ranked, and topic relevance scores are output; the two types of qualification standard texts are input into the neural re-ranking model. The relevance scores of the eligibility criteria are ranked and output as eligibility criteria relevance scores. The re-ranked scores are then combined with the scores of the first two criteria and ranked to screen samples to be reviewed. Here, k is the model version number. S3. Use the scores and rankings of the samples to be reviewed as auxiliary prior information, and input them along with the corresponding patient-trial samples into the large language model for clinical review. Output a structured review result including review score, review label and clinical dimension review information. S4. Based on the true matching labels of the samples to be reviewed, the scores and rankings of each item, and the structured review results, a feedback record is constructed, and the feedback diagnostic agent determines the error type and deviation direction to obtain effective diagnostic results. S5. Based on the deviation distribution of effective diagnostic results, the decision control agent selects an update mode of strict matching update, relaxed matching update, parameter modification or unchanged, and generates an execution specification that limits the updatable configuration items. The first three update modes are used to add predefined clinical input fields and retrain the model, reduce predefined unnecessary clinical input fields and retrain the model, and adjust the score fusion coefficient and / or matching judgment threshold without retraining. S6. The updating agent generates the updated neural reordering model based on the execution specification. And compared by the verifying agent and Based on the corresponding evaluation metrics on the same fixed validation set, acceptance is determined. , roll back to Alternatively, manual verification can be performed, and the selected model can output the patient-trial matching score and predicted matching label.

[0031] In this embodiment, matching is first performed using a neural reordering model, followed by clinical verification using a large model. This approach balances matching efficiency, semantic accuracy, and clinical interpretability. The output of the neural reordering model is then used as an auxiliary prior for the large model verification. This design allows the large model to utilize the initial ranking information provided by the neural model while also performing independent verification based on original clinical evidence. Finally, multiple agents collaborate to complete error diagnosis, update strategy selection, minimal code modification, and post-modification verification, greatly improving the automation, controllability, and auditability of the model optimization process.

[0032] In this embodiment, "patient-trial sample" refers to a matching group consisting of a patient and a clinical trial. True match label. Used to represent the true match between patients and clinical trials. =1 indicates a positive example, meaning the patient is a match for the clinical trial. =0 indicates a negative case, meaning the patient does not match the clinical trial; true matching labels can come from confirmed enrollment results, annotation results from clinical researchers, or pre-built labeled datasets, and are used as supervisory information during training and validation.

[0033] Furthermore, patient data may include patient identification, age, gender, body mass index (BMI), blood pressure, chief complaint, present illness, past medical history, diagnosis, examination diagnosis or conclusion, examination findings, examination site, and medication history. Missing fields may be retained using null flags or preset missing flags so that the model can distinguish between "clearly not satisfied" and "evidence not provided." Clinical trial data may include trial identification, trial name, trial summary, related or target disease, inclusion criteria, exclusion criteria, and investigational drug information.

[0034] The patient subject text refers to the text used to characterize the patient's primary disease subject, and can be obtained by combining at least one of the following: diagnosis, examination diagnosis or examination conclusion, present medical history, and chief complaint. The clinical trial subject text refers to the text used to characterize the target disease subject of the clinical trial, and can be obtained by combining at least one of the following: trial name, trial abstract, and related diseases.

[0035] The patient eligibility criteria text refers to the text of clinical evidence from patients used to make corresponding judgments with the trial inclusion criteria, exclusion criteria, and investigational drug restrictions. It can be obtained by combining at least one of the following: diagnosis, examination conclusions, examination findings, present illness, chief complaint, past medical history, medication history, age, sex, BMI, blood pressure, and examination site. The clinical trial eligibility criteria text can be obtained by combining at least one of the following: inclusion criteria, exclusion criteria, and investigational drug information.

[0036] Model version Associated with version number k. A model version must save at least: network parameters or checkpoints for the topic relevance sub-model, network parameters or checkpoints for the qualification criterion relevance sub-model, a list of current input fields, score fusion coefficients, matching thresholds, data partitioning files, training configuration, and evaluation results. Updated version. Generated in a separate storage location, without overwriting. .

[0037] Specifically, in step S1, a patient-trial sample with real-matching labels is obtained, including the subject text and eligibility criteria text of the patient and the clinical trial, respectively.

[0038] Let the clinical trial set and the patient set be respectively: in, Indicates the first One clinical trial, Number of clinical trials; Indicates the first One patient, For the number of patients.

[0039] For each patient-test sample Construct the corresponding real matching tags ∈{0,1}. Where, the parameter... subscript The corresponding clinical trial index and patient index are used, and all other related parameters in the following text adopt this representation. Training, validation, and testing data are divided into trial-level categories according to the clinical trial identifier (trial_id) to avoid information leakage caused by the same clinical trial appearing in both the training and testing sets.

[0040] In one example, the ratio of training, validation, and test sets is 6:1:3, and the random seed is 42. After the initial split, a fixed split file is saved, which is reused for topic relevance ranking, eligibility criterion relevance ranking, and validation of each model version. Feedback diagnostics, update decisions, and candidate model validation use only the training or validation set. The test set remains invisible during the feedback optimization process and is only used for the final performance evaluation.

[0041] When concatenating text, you can add field names or field identifiers before each field and separate them with delimiters to preserve the source of the information. For numerical conditions such as age, BMI, blood pressure, or test results, you can retain the original values ​​and their units; for negative expressions, you can retain their context and not delete negative words such as "not seen," "deny," or "none" from the original sentence.

[0042] In step S2, the two types of topic samples are input into the neural reordering model. The topic relevance is ranked, and topic relevance scores are output; the two types of qualification standard texts are input into the neural re-ranking model. The relevance scores of the eligibility criteria are ranked and the relevance scores of the eligibility criteria are output. The re-ranking scores are then combined with the scores of the first two criteria to filter the samples to be reviewed. Here, k is the model version number.

[0043] Neural Reordering Model This is used for topic relevance processing and eligibility criterion relevance processing. Both processing methods can employ a Transformer cross-encoder structure, which constructs text pairs from clinical trial text and patient text, obtains matching representations using joint encoding, and outputs a single raw score logit from the classification head, which is then mapped to the [0,1] interval by the sigmoid activation function.

[0044] In one example, the Chinese RoBERTa is used as the pre-trained encoder, with text pairs in the form of "[CLS]clinical trial text[SEP]patient text[SEP]". The classification head includes a Dropout layer and a linear layer. The Dropout probability is 0.1, and the linear layer outputs a single raw score logit. The training objective uses binary cross-entropy loss with logit input (BCEWithLogitsLoss). The input is encoded by a word segmenter with a maximum length of 512, and padding and truncation are performed. RoBERTa is an existing pre-trained language model optimized based on BERT.

[0045] Topic relevance ranking uses clinical trial topic texts and patient topic texts, and its output topic relevance score can be expressed as follows: A higher value indicates a greater relevance between the patient's disease topic and the target disease of the clinical trial. The topic relevance binary classification loss can be expressed as: in, Indicates the experiment With patients The topic relevance score.

[0046] The eligibility criterion relevance ranking uses clinical trial eligibility criterion texts and patient eligibility criterion texts, and its output eligibility criterion relevance score can be expressed as follows: A higher value indicates that the patient is more likely to meet the eligibility requirements for clinical trial enrollment. The binary loss related to eligibility criteria can be expressed as: in, Indicates the experiment With patients The relevance score of the qualification criteria.

[0047] In one example, the topic relevance sub-model is trained for 3 epochs, and the Transformer main learning rate is 2×10. -5 The learning rate for the linear layer is 1×10. -3 The early cessation patience value is 1; the eligibility criterion relevance submodel is initialized with the checkpoint of the topic relevance submodel, and trained for 5 more rounds on the clinical trial eligibility criterion text and patient eligibility criterion text, with a Transformer subject learning rate of 5×10. -6 The learning rate for the linear layer is 1×10. -3The early stopping patience value is 2. During training, the validation set loss is used as the basis for model selection, and the model parameters with the lowest validation loss are saved; if the validation set is unavailable, the training loss can be used as a fallback reference. The training optimizer uses AdamW, and grouped learning rates are applied to the Transformer body and the classification head.

[0048] In the above example, the training batch size of the data module is configured to 16, and the training data loader uses half of this configuration, so the actual training batch size is 8; the batch size for the validation, testing, and scoring phases is 16. The model base, number of training epochs, learning rate, batch size, and early stopping parameters described above are all implementation examples and are not intended to limit the scope of the claims.

[0049] After obtaining the two scores, the re-ranking score is calculated using the following weighted fusion formula: in, The score fusion coefficient, 0 ≤ ≤1. The eligibility criterion relevance score reflects whether a patient meets the eligibility criteria for clinical trial enrollment, while the topic relevance score preserves the matching information between the patient's disease topic and the target disease of the trial. In one example, =0.7.

[0050] For each clinical trial It can also rank the corresponding patient candidates in descending order according to the reordering function, and finally output the result. , , and The samples to be reviewed can be obtained according to preset rankings, preset proportions, or screening rules corresponding to the recruitment task; in one example, the top 40% of patients in each clinical trial are selected as the samples to be reviewed to avoid calling the large language model for all patients.

[0051] Furthermore, when it is necessary to output predicted matching labels, the matching judgment threshold of the current version can be used. Perform the following judgment: in, For indicator functions; For predicting label symbols, represent the neural reordering model. Output of the experiment With patients The predicted matching label only takes two values. It was determined to be a match. The result was determined to be a mismatch.

[0052] In step S3, the scores and rankings of the samples to be reviewed are used as auxiliary prior information, and together with the corresponding patient-trial samples, they are input into the large language model for clinical review, and the output is a structured review result including review score, review label and clinical dimension review information.

[0053] For each sample to be reviewed, the patient-trial original text and auxiliary prior are used to construct a prompt message. The auxiliary prior can be represented as: in, Indicating clinical trials With patients Auxiliary prior information.

[0054] Furthermore, the auxiliary prior information can be converted into the following natural language: "This patient's ranking among current clinical trial candidates is { } name, topic relevance score is { The relevance score of the qualification criteria is { The reordered score after fusion is { }".

[0055] Furthermore, the prompt may also include a review instruction: "Judgment should be made independently based primarily on the original clinical trial text and the original patient medical records. The above scores and rankings are only supplementary evidence and should not be mechanically applied to the conclusions of the preceding model." In other words, this review instruction instructs the large language model to use the patient-trial sample as the primary basis for judgment and the aforementioned supplementary prior information as supplementary evidence.

[0056] The large language model performs judgments according to a preset review order: first, it extracts the target disease or indication of the clinical trial; second, it extracts the patient's primary diagnosis; third, it determines whether the two belong to the same disease domain or have a direct medical relevance; if the disease domains match, it continues to assess the satisfaction of inclusion criteria and the risk of hitting exclusion criteria; finally, it provides a review score, review label, and a conclusion indicating whether the patient agrees with the neural reordering model's judgment. The final structured review results include a review score, review label, and clinical dimension review information. Further, the clinical dimension review information includes disease matching degree, inclusion criterion satisfaction degree, and exclusion risk degree; wherein, the disease matching degree characterizes the degree of association between the patient's primary diagnosis and the target disease of the clinical trial, the inclusion criterion satisfaction degree characterizes the degree to which the patient's clinical information meets the clinical trial inclusion criteria, and the exclusion risk degree characterizes the degree of risk that the patient will hit the clinical trial exclusion criteria.

[0057] In one example, when the target disease of the clinical trial does not significantly match the patient's primary diagnosis, a review label of 0 and a review score ≤ 0.1000 are set. Evaluation of inclusion criteria fulfillment and exclusion risk is only continued when the disease domains match. All numerical outputs are constrained to the [0,1] interval after parsing.

[0058] The structured review result can be represented as: in, For review of scores; For verification labels; For disease matching degree; To include the standard satisfaction level; To eliminate risk; This indicates whether the large language model agrees with the neural reordering model's judgment. These are the review reasons that can be interpreted using natural language; the superscript LLM indicates a large language model, and the subscript ij indicates the clinical trial corresponding to each review information. With patients Optionally, the structured review results can be output as a JSON object or other parsable structured data object.

[0059] In one example, the large language model uses Qwen / Qwen3-8B and is invoked through a service compatible with the OpenAI ChatCompletions interface; temperature is set to 0.0, top_p is set to 0.95, and the output format is constrained by response_format={"type":"json_object"}. The maximum length parameter is configured according to the context length and output length supported by the invoked interface. This model name and interface configuration are only examples and do not limit the specific type or deployment method of the large language model. A structured output object may include the following fields: score, label, disease_match_score, inclusion_match_score, exclusion_risk_score, agree_with_rerank_model, and reason. Among them, disease_match_score, inclusion_match_score, and exclusion_risk_score represent the disease matching score, inclusion criterion satisfaction score, and exclusion risk score, respectively. The parsing program checks whether the fields are complete, whether the field types conform to the convention, and whether the values ​​fall within the [0,1] range. When parsing fails, the failure status can be recorded and the sample can be transferred for re-verification or manual inspection, rather than directly using the results that fail the structural verification as a valid diagnostic basis.

[0060] It should be noted that the neural reordering score is retained in the final recommendation output. and large model verification scores and review label The large-scale model review results serve as clinical semantic-level review opinions, assisting in the final matching judgment. In the final recommendation ranking, the system uses the neural re-ranking results as the initial ranking basis, and outputs the large-scale model review score and review label together as clinical review opinions.

[0061] In step S4, a feedback record is constructed based on the true matching labels of the sample to be reviewed, the scores and rankings of each item, and the structured review results. The feedback diagnostic agent then determines the error type and deviation direction to obtain an effective diagnostic result.

[0062] For each patient awaiting review - a feedback record is constructed: in, The sample information should include at least patient and clinical trial identifiers. Indicates a true match tag; This represents the output of the neural re-ranking model, including topic relevance score, eligibility criterion relevance score, re-ranking score, and ranking. The structured review results generated by the large language model include review score, review label, disease matching degree, inclusion criterion satisfaction degree, exclusion risk degree, and optional consent markers and review reasons; This includes the trial name, abstract, related diseases, inclusion criteria, exclusion criteria, and information on the investigational drug; This includes diagnosis, examination results, examination findings, present medical history, chief complaint, past medical history, medication history, age, gender, BMI, blood pressure, and examination site.

[0063] Optionally, the feedback log can also save the model version number, data partition identifier, prompt template version, review time, and parsing status. This allows for tracing diagnostic results back to the corresponding input, model, and prompt configurations, providing a unified data foundation for subsequent error diagnosis and strategy decisions. These traceability fields do not alter the trial-patient matching judgment itself.

[0064] Based on this, in an optional implementation, the feedback diagnostic agent receives feedback records and performs sample-level error diagnosis, that is, the feedback diagnostic agent determines the error type and deviation direction to obtain effective diagnostic results, including: When the true matching label is a negative example and meets any of the following conditions: the re-ranking score is not lower than the first high score threshold, or the re-ranking score is not lower than the second high score threshold and the review label is a negative example, the error type is determined to be over-inclusion error and the deviation direction is enhanced matching constraint. When the actual matching label is a positive example and meets any of the following conditions: the re-ranking score is not higher than the first low score threshold, or the re-ranking score is not higher than the second low score threshold and the review label is a positive example, the error type is determined to be an over-exclusion error and the deviation direction is relaxed matching constraints. When the re-ranking score is within the preset boundary range, and the absolute value of the difference between the review score and the re-ranking score reaches the preset difference, the error type is determined to be calibration error and the deviation direction is parameter calibration. If the aforementioned conditions are not met, the error type is determined to be no error and the deviation direction is determined to be no update required; Generate a diagnostic result that includes the error type, the deviation direction, and the diagnostic confidence level; Diagnostic results with a confidence level lower than a preset confidence threshold are filtered out, and the remaining diagnostic results are taken as valid diagnostic results. Wherein, positive examples indicate that the patient is a match for the clinical trial, negative examples indicate that the patient is not a match for the clinical trial, the first high score threshold is greater than the second high score threshold, and the first low score threshold is less than the second low score threshold.

[0065] Specifically, the diagnostic agent first checks whether the re-ranking score conforms to the current version of the fusion rules and whether there is a significant discrepancy between the topic relevance score and the eligibility criterion relevance score; then, it combines the true match label, review score, review label, and clinical dimension review information to determine the main error type C and the bias direction D. in, The overinclusion error indicates that the neural reordering model assigns excessively high scores or biases positive judgments to true negative examples, with the bias directed towards reinforcing the matching constraints. . The over-exclusion error indicates that the neural reordering model gives too low a score or a biased negative judgment to true positive examples, with the bias pointing towards relaxing the matching constraints. . This indicates that the main deviation comes from the fusion coefficient or the matching decision threshold, and the direction of the deviation is from the parameter calibration. . No update is required, meaning no update will be triggered. .

[0066] Let the first high score threshold be H1, the second high score threshold be H2, the first low score threshold be L1, and the second low score threshold be L2, where H1>H2 and L1<L2. The diagnosis sequence may be executed in the order of over-inclusion, over-exclusion, calibration, and no error, so that each sample is assigned a primary error type.

[0067] When =0 and ≥H1, it is determined as an over-inclusion error; or, when =0, ≥H2 and =0, it is determined as an over-inclusion error.

[0068] When =1 and ≤L1, it is determined as an over-exclusion error; or, when =1, ≤L2 and =1, it is determined as an over-exclusion error.

[0069] When the aforementioned over-inclusion and over-exclusion conditions are not met, if falls within the preset boundary interval [B min ,B max , reaches the preset difference , it is determined as a calibration error. is configured by the current version and determined on fixed validation data, and is used to exclude invalid discrepancies caused by numerical rounding or slight fluctuations.

[0070] In an example, H1=0.70, H2=0.55, L1=0.30, L2=0.45, and the preset boundary interval is [0.30, 0.70]. The preset difference may be pre-configured according to the score distribution on the validation set; when the determined value is not disclosed, a specific numerical value is not regarded as a necessary limitation.

[0071] The diagnosis result output by the diagnosis agent may include the error type , deviation direction , error label , diagnosis reason , support signal , diagnosis confidence and abandonment flag : Wherein, the support signal This may include triggering threshold rules, the consistency between the true label and the review label, disease matching degree, satisfaction of inclusion criteria, exclusion risk degree, and relevant clinical evidence fields. Diagnostic confidence. The value is [0,1]. The diagnostic agent can output the diagnostic confidence level using constraint prompts, and the program side verifies its range and required fields; when the diagnostic basis is insufficient, the structure verification fails, or the diagnostic confidence level is lower than the preset confidence threshold, an abandonment flag is set.

[0072] In one example, the preset confidence threshold is 0.45. Results with a diagnostic confidence level below 0.45 are filtered out, while the remaining results that pass structural validation and do not have an abandonment flag are considered valid diagnostic results.

[0073] In step S5, the decision control agent selects an update mode based on the deviation distribution of the effective diagnostic results: strict matching update, relaxed matching update, parameter modification, or unchanged update mode, and generates an execution specification that limits the updatable configuration items. The first three update modes are used to add predefined clinical input fields and retrain the model, reduce predefined unnecessary clinical input fields and retrain the model, and adjust the score fusion coefficient and / or matching judgment threshold without retraining.

[0074] The decision control agent aggregates all valid diagnostic results. In one optional implementation, based on the deviation distribution of the valid diagnostic results, the decision control agent selects an update mode that includes strict matching update, relaxed matching update, parameter modification, or no change. The number of valid diagnostic results corresponding to overinclusion error, overexclusion error and calibration error, the proportion of each of these numbers to all valid diagnostic results and the corresponding average diagnostic confidence were statistically analyzed. When the number of valid diagnostic results corresponding to calibration errors, their proportion of all valid diagnostic results, and the average diagnostic confidence, as well as the ratio of the larger of the number of valid diagnostic results corresponding to calibration errors to the number of valid diagnostic results corresponding to over-inclusion errors and over-exclusion errors, reach the corresponding preset threshold, it is determined that there is a calibration deviation and the parameter is modified. When the triggering conditions for parameter modification are not met, if the number of valid diagnostic results corresponding to the over-exclusion error reaches the preset minimum number, is greater than the number of valid diagnostic results corresponding to the over-inclusion error, and the preset field number lower limit is not triggered, the relaxed matching update is selected. When the triggering conditions for parameter modification are not met, if the number of valid diagnostic results corresponding to over-inclusion errors reaches the preset minimum number and is greater than the number of valid diagnostic results corresponding to over-exclusion errors, the strict matching update mode is selected. If none of the aforementioned update mode trigger conditions are met, the option is to remain unchanged.

[0075] Specifically, effective diagnostic results include over-inclusion errors, over-exclusion errors, calibration errors, and error-free diagnoses that were not filtered by confidence levels and did not have a drop-off flag. Let the number of effective diagnoses for over-inclusion errors, over-exclusion errors, and calibration errors be respectively... , and The total number of effective diagnoses is The corresponding ratio is , and The corresponding average diagnostic confidence levels are respectively , and When the number of errors of a certain type is 0, its average diagnostic confidence is recorded as 0; When =0, simply choose to keep it unchanged.

[0076] The update mode M is: Among them, KEEP, STRICT_UPDATE, LOOSE_UPDATE, and CALIBRATION_UPDATE represent keeping the same, strict matching update, relaxed matching update, and parameter modification, respectively.

[0077] The parameter modification CALIBRATION_UPDATE has a priority order. When Reaching the minimum calibration threshold Reaching the calibration ratio threshold The calibrated average diagnostic confidence threshold was reached, and When the calibration relative dominance threshold is reached, a calibration deviation is identified, and parameter modifications are selected. The calibration evidence ratio is the number of calibration diagnostics with preset calibration support signals and... The ratio; the preset calibration support signal may include the difference between the review score and the reordering score reaching The criteria include at least one of the following: discrepancies between the verification label and the original matching determination, and the structured verification field passing the integrity check. In a further constrained embodiment, the proportion of calibration evidence is also required to reach a preset threshold.

[0078] When the parameter modification trigger condition is not met, Reaching the minimum update threshold > And if the lower limit for the number of fields is not triggered, choose to relax the matching update LOOSE_UPDATE; when Reaching the minimum update threshold and > In the case of strict matching, STRICT_UPDATE is selected; otherwise, KEEP is selected to remain unchanged.

[0079] In one example, the calibration trigger conditions include ≥45、 ≥0.50, Calibration evidence ratio ≥0.60 ≥0.55 and ≥1.40; the minimum number of updates for both strict matching and relaxed matching is 45. The denominator uses... ,in This indicates the operator for taking the larger value, used in... and Avoid dividing by zero when all values ​​are 0.

[0080] After selecting an update mode, the decision-making control agent generates a structured execution specification. This execution specification includes at least: the baseline model version, the update mode, allowed configuration items, prohibited modification items, a list of candidate input fields or parameter candidates, whether retraining is required, the baseline result path, candidate result paths, a fixed validation set identifier, and validation conditions.

[0081] Let the available clinical field set be Fall, the current version's input field set be Fk, the predefined set of non-essential clinical fields be Foptional, and the set of core fields that must be retained be Fcore. During strict matching updates, only fields specified by the execution guidelines are allowed to be added from Fall\Fk. Added fields can prioritize inclusion criteria, exclusion criteria, medication history, past medical history, examination results, and other fields related to eligibility constraints and exclusion risks. During relaxed matching updates, only fields specified by the execution guidelines are allowed to be deleted from Fk∩Foptional, and Fcore must remain included in the updated field set. If deletion would cause the number of fields to fall below a preset lower limit or would delete core fields, the update should be blocked, and the option to leave it unchanged or perform manual verification should be selected. Here, \ represents set difference operation, and ∩ represents intersection operation.

[0082] When parameters are modified, the execution specification only allows modification of the score fusion coefficient. and / or matching threshold The parameter candidate values ​​are given by a pre-configured allowed range or candidate list. The agent performing the update must not expand this range, change the parameters of the Transformer main network, or trigger retraining.

[0083] In step S6, the executing agent generates an updated neural reordering model based on the execution specification. And compared by the verifying agent and Based on the corresponding evaluation metrics on the same fixed validation set, acceptance is determined. , roll back to Alternatively, manual verification can be performed, and the selected model can output the patient-trial matching score and predicted matching label.

[0084] In an optional implementation, the updated neural reordering model is generated by the execution updating agent based on the execution specification. ,include: Backup neural reordering model when the selected update mode is not immutable. The corresponding benchmark programs and benchmark models are generated and stored separately as candidate programs and candidate models; Only modify the input fields, fusion coefficients, matching thresholds and / or result saving paths specified in the execution specifications, while keeping the training process, evaluation logic and irrelevant data streams unchanged; Under strict matching or relaxed matching updates, retrain the candidate model according to the modified input fields; under parameter modifications, maintain the neural reordering model. The network parameters remain unchanged, and the score fusion coefficient and / or matching judgment threshold are adjusted. Running the candidate program and the candidate model yields the updated neural reordering model. And its evaluation results.

[0085] Specifically, the agent executing the update forms a modification task according to the structured execution specification, and is only allowed to modify the list of input fields and the score fusion coefficients. Matching threshold And / or the result storage path. The training process, data partitioning files, evaluation logic, and data flows unrelated to this round of updates remain unchanged. Candidate modifications can be generated by first calling the code generation model, and then the scope of modifications can be verified through whitelist path checks, configuration item checks, and difference checks. When the code generation results do not meet the constraints, the specified configuration items can be modified using a local exact replacement method.

[0086] In both strict matching and relaxed matching update modes, the candidate program reconstructs the corresponding text based on the updated set of input fields and retrains the candidate neural reordering model to obtain... In parameter modification mode, reuse The network parameters are used to recalculate the re-ranking score based only on the candidate fusion coefficients and / or regenerate the predicted matching label based on the candidate matching judgment threshold. The corresponding configuration and network parameters are registered together as... .

[0087] After execution, save the candidate program, candidate model, running command or task description, modification prompts, configuration differences, standard output, standard error, running summary, and evaluation results. The updated agent checks whether the target replacement items are completed, whether prohibited modifications remain unchanged, whether the candidate program can be loaded, whether the model output fields are complete, and whether the result path is isolated from the baseline path; only candidate results that pass verification enter the validation phase.

[0088] Furthermore, verify the agent's reading and Evaluation reports are taken from the same fixed validation set, and the metrics with the same names and comparable characteristics from the two reports are taken as a common evaluation metric set Icommon. In an optional implementation, the comparison is performed by the validation agent. and The corresponding evaluation metrics on the same fixed validation set include: Sure and The evaluation results contain common evaluation indicators, which are used as public evaluation indicators. Calculate the absolute and relative changes of each public evaluation indicator according to the pre-defined performance evaluation criteria; if When the corresponding public evaluation indicator value is zero, only the absolute change of that public evaluation indicator is calculated; When the public assessment indicators that meet the first preset ratio improve and the degradation of the remaining public assessment indicators does not exceed the first permissible range, acceptance is granted. ; When a public assessment indicator that has reached the second preset proportion degrades and the degree of degradation exceeds the second allowable range, it will revert to [a previous setting]. ; In all other cases, manual verification is performed, and the evaluation results of both models are retained until the manual verification is completed, with the neural reordering model being used. As the default selected model.

[0089] Specifically, a fixed validation set can be used for consistency verification by partitioning the data into file identifiers, sample identifier sets, and / or file digests. For each common evaluation metric m, let the benchmark metric value be... Candidate indicator value Performance Direction Factor The value is 1 when a larger indicator is better, and -1 when a smaller indicator is better. The absolute change after directional normalization is calculated using the following formula. : when When ≠0, calculate the relative change. : when When =0, only absolute changes are retained. Relative changes are not calculated. A value greater than 0 indicates that the candidate model improves on metric m. <0 indicates that the candidate model degenerates on index m; for degenerate indices, the magnitude of degradation can be determined by - Or it can be determined according to the absolute change scale specified in the implementation guidelines.

[0090] Candidate models are not automatically accepted and are retained if the candidate program fails to execute, evaluation results are missing, consistency verification on the fixed validation set fails, or no comparable public evaluation metrics exist. It will then proceed to manual inspection or failure processing.

[0091] Verification mode V includes acceptance rollback and manual inspection : When the proportion of improved public assessment indicators to the total number of public assessment indicators reaches a first preset proportion, and the degradation of the remaining public assessment indicators does not exceed a first permissible range, select ACCEPT and accept. In one example, the first preset ratio is 75%, and the first allowable range is 5%.

[0092] When the proportion of degraded public assessment indicators to the total number of public assessment indicators reaches the second preset proportion, and the degree of degradation exceeds the second allowable range, select ROLLBACK and revert to the previous state. In one specific embodiment, the second preset ratio is 60%, and the second allowable range is 7%.

[0093] If the acceptance and rollback conditions are not met, select REVIEW and proceed to manual inspection. Keep the process open until manual inspection is complete. and The evaluation report, detailed indicators, procedures, and models are still based on the default settings. As the selected model, it avoids automatically replacing the stable version with uncertain candidate updates.

[0094] The verification agent outputs the verification mode, details of common indicators, absolute changes, relative changes, selected model version, and selected result source. If ACCEPT is selected, k can be updated to k+1 and the feedback process can be re-executed in subsequent batches; if ROLLBACK or REVIEW is selected, the baseline version number remains unchanged. This embodiment prioritizes performing only one round of feedback update to verify the update mechanism. Of course, multiple iterations can also be performed as needed, provided that each round passes fixed verification and version management.

[0095] In the final recommendation ranking, the system uses the neural re-ranking results as the initial ranking basis, and outputs the large model review score and review label as clinical review opinions.

[0096] Based on the same inventive concept, embodiments of the present invention also provide an intelligent patient matching system based on multi-stage recommended clinical trials, including: The sample acquisition module is used to acquire patient-trial samples with real-match labels, including the subject text and eligibility criteria text of each patient and clinical trial. The reordering module is used to input two types of topic samples into the neural reordering model. The topic relevance is ranked, and topic relevance scores are output; the two types of qualification standard texts are input into the neural re-ranking model. The relevance scores of the eligibility criteria are ranked and output as eligibility criteria relevance scores. The re-ranked scores are then combined with the scores of the first two criteria and ranked to screen samples to be reviewed. Here, k is the model version number. The large language model review module is used to take the scores and rankings of the samples to be reviewed as auxiliary prior information, and input them along with the corresponding patient-trial samples into the large language model for clinical review. The output is a structured review result including review score, review label and clinical dimension review information. The feedback record construction and feedback diagnosis module is used to construct feedback records based on the true matching labels of the sample to be reviewed, the scores and rankings of each item, and the structured review results. The feedback diagnosis agent determines the error type and deviation direction to obtain effective diagnostic results. The decision control module is used by the decision control agent to select an update mode of strict matching update, relaxed matching update, parameter modification or no change based on the deviation distribution of effective diagnostic results, and generate an execution specification that limits the updatable configuration items. The first three update modes are used to add predefined clinical input fields and retrain the model, reduce predefined unnecessary clinical input fields and retrain the model, and adjust the score fusion coefficient and / or matching judgment threshold without retraining. The execution update and verification module is used by the execution update agent to generate the updated neural reordering model based on the execution specification. And compared by the verifying agent and Based on the corresponding evaluation metrics on the same fixed validation set, acceptance is determined. , roll back to Alternatively, manual verification can be performed, and the selected model can output the patient-trial matching score and predicted matching label.

[0097] In this embodiment, matching is first performed using a neural reordering model, followed by clinical verification using a large model. This approach balances matching efficiency, semantic accuracy, and clinical interpretability. The output of the neural reordering model is then used as an auxiliary prior for the large model verification. This design allows the large model to utilize the initial ranking information provided by the neural model while also performing independent verification based on original clinical evidence. Finally, multiple agents collaborate to complete error diagnosis, update strategy selection, minimal code modification, and post-modification verification, greatly improving the automation, controllability, and auditability of the model optimization process.

[0098] In an optional implementation, the topic relevance score, eligibility criterion relevance score, re-ranking score, and ranking of the sample to be reviewed are converted into auxiliary prior information in natural language form, and the auxiliary prior information is written into the prompt information of the input large language model; The prompt information includes a verification instruction that instructs the large language model to use the patient-trial sample as the primary basis for judgment and the auxiliary prior information as supplementary evidence.

[0099] In an optional implementation, the clinical dimension review information includes disease matching degree, inclusion criterion satisfaction degree, and exclusion risk degree, and the structured review results are output as parsable structured data objects; The disease matching degree represents the degree of association between the patient's primary diagnosis and the target disease of the clinical trial; the inclusion criterion satisfaction degree represents the degree to which the patient's clinical information meets the clinical trial inclusion criteria; and the exclusion risk degree represents the degree of risk that the patient will meet the clinical trial exclusion criteria.

[0100] In an optional implementation, the step of determining the error type and deviation direction by the feedback diagnostic agent to obtain an effective diagnostic result includes: When the true matching label is a negative example and meets any of the following conditions: the re-ranking score is not lower than the first high score threshold, or the re-ranking score is not lower than the second high score threshold and the review label is a negative example, the error type is determined to be over-inclusion error and the deviation direction is enhanced matching constraint. When the actual matching label is a positive example and meets any of the following conditions: the re-ranking score is not higher than the first low score threshold, or the re-ranking score is not higher than the second low score threshold and the review label is a positive example, the error type is determined to be an over-exclusion error and the deviation direction is relaxed matching constraints. When the re-ranking score is within the preset boundary range, and the absolute value of the difference between the review score and the re-ranking score reaches the preset difference, the error type is determined to be calibration error and the deviation direction is parameter calibration. If the aforementioned conditions are not met, the error type is determined to be no error and the deviation direction is determined to be no update required; Generate a diagnostic result that includes the error type, the deviation direction, and the diagnostic confidence level; Diagnostic results with a confidence level lower than a preset confidence threshold are filtered out, and the remaining diagnostic results are taken as valid diagnostic results. Wherein, positive examples indicate that the patient is a match for the clinical trial, negative examples indicate that the patient is not a match for the clinical trial, the first high score threshold is greater than the second high score threshold, and the first low score threshold is less than the second low score threshold.

[0101] In an optional implementation, the decision-control agent selects an update mode—strict matching update, relaxed matching update, parameter modification, or unchanged—based on the deviation distribution of effective diagnostic results, including: The number of valid diagnostic results corresponding to overinclusion error, overexclusion error and calibration error, the proportion of each of these numbers to all valid diagnostic results and the corresponding average diagnostic confidence were statistically analyzed. When the number of valid diagnostic results corresponding to calibration errors, their proportion of all valid diagnostic results, and the average diagnostic confidence, as well as the ratio of the larger of the number of valid diagnostic results corresponding to calibration errors to the number of valid diagnostic results corresponding to over-inclusion errors and over-exclusion errors, reach the corresponding preset threshold, it is determined that there is a calibration deviation and the parameter is modified. When the triggering conditions for parameter modification are not met, if the number of valid diagnostic results corresponding to the over-exclusion error reaches the preset minimum number, is greater than the number of valid diagnostic results corresponding to the over-inclusion error, and the preset field number lower limit is not triggered, the relaxed matching update is selected. When the triggering conditions for parameter modification are not met, if the number of valid diagnostic results corresponding to over-inclusion errors reaches the preset minimum number and is greater than the number of valid diagnostic results corresponding to over-exclusion errors, the strict matching update mode is selected. If none of the aforementioned update mode trigger conditions are met, the option is to remain unchanged.

[0102] In an optional implementation, the updated neural reordering model is generated by the execution updating agent based on the execution specification. ,include: Backup neural reordering model when the selected update mode is not immutable. The corresponding benchmark programs and benchmark models are generated and stored separately as candidate programs and candidate models; Only modify the input fields, fusion coefficients, matching thresholds and / or result saving paths specified in the execution specifications, while keeping the training process, evaluation logic and irrelevant data streams unchanged; Under strict matching or relaxed matching updates, retrain the candidate model according to the modified input fields; under parameter modifications, maintain the neural reordering model. The network parameters remain unchanged, and the score fusion coefficient and / or matching judgment threshold are adjusted. Running the candidate program and the candidate model yields the updated neural reordering model. And its evaluation results.

[0103] In an optional implementation, the comparison by the verification agent... and The corresponding evaluation metrics on the same fixed validation set include: Sure and The evaluation results contain common evaluation indicators, which are used as public evaluation indicators. Calculate the absolute and relative changes of each public evaluation indicator according to the pre-defined performance evaluation criteria; if When the corresponding public evaluation indicator value is zero, only the absolute change of that public evaluation indicator is calculated; When the public assessment indicators that meet the first preset ratio improve and the degradation of the remaining public assessment indicators does not exceed the first permissible range, acceptance is granted. ; When a public assessment indicator that has reached the second preset proportion degrades and the degree of degradation exceeds the second allowable range, it will revert to [a previous setting]. ; In all other cases, manual verification is performed, and the evaluation results of both models are retained until the manual verification is completed, with the neural reordering model being used. As the default selected model.

[0104] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program causes a computer to execute the intelligent matching method for patients in multi-stage recommended clinical trials provided in any embodiment of the present invention.

[0105] In embodiments of the present invention, any combination of one or more storage media may be used. The storage medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0106] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 3 The diagram shown is a structural schematic of the electronic device 100 provided in an embodiment of the present invention. In some embodiments, the electronic device may be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), or other terminal device. Furthermore, the intelligent matching method for patients in multi-stage recommended clinical trials provided in this embodiment of the present invention can also be applied to databases, servers, and service response systems based on terminal artificial intelligence. This embodiment of the present invention does not limit the specific application scenarios of the intelligent matching method for patients in multi-stage recommended clinical trials.

[0107] like Figure 3 As shown, the electronic device 100 provided in this embodiment of the invention includes a memory 101 and a processor 102.

[0108] The memory 101 is used to store computer programs; preferably, the memory 101 includes various media that can store program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.

[0109] Specifically, memory 101 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Electronic device 100 may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 101 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0110] The processor 102 is connected to the memory 101 and is used to execute the computer program stored in the memory 101 so that the electronic device 100 executes the intelligent matching method for patients based on multi-stage recommended clinical trials provided in any embodiment of the present invention.

[0111] In an optional implementation, the processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, evaluation index gate or transistor logic devices, or evaluation index hardware components.

[0112] In an optional embodiment, the electronic device 100 of the present invention may further include a display 103. The display 103 is communicatively connected to the memory 101 and the processor 102, and is used to display a GUI interactive interface related to a patient intelligent matching method based on a multi-stage recommendation clinical trial.

[0113] It is understood that the intelligent matching system, storage medium, and electronic device for patients in multi-stage recommended clinical trials provided in the embodiments of the present invention correspond to the intelligent matching method for patients in multi-stage recommended clinical trials provided in the embodiments of the present invention. The explanations, examples, and beneficial effects of the relevant contents can be referred to the corresponding parts of the method, and will not be repeated here.

[0114] In summary, compared with existing technologies, it has the following beneficial effects: 1. Improve the accuracy of patient matching in clinical trials This invention combines the semantic ranking capabilities of a neural model with the clinical verification capabilities of a large language model through a hierarchical process of "neural reordering—large model verification." The neural reordering stage learns the semantic relationships between the experimental text and patient medical records; the large model verification stage makes clinical judgments from the perspectives of disease matching, fulfillment of inclusion criteria, and exclusion risks. Compared to methods relying solely on keywords or single model scores, this invention can more precisely distinguish different types of candidate patients, thereby improving the final matching accuracy.

[0115] 2. Improve the interpretability of matching results The large-scale model review module in this invention requires the output of structured clinical judgment results, rather than a single matching conclusion. The system can output not only the final matching score and recommendation label, but also the disease matching degree, the degree of fulfillment of inclusion criteria, exclusion risk, whether the neural reordering model's judgment is agreed upon, and natural language reasons. Therefore, when a patient is recommended, the system can explain the basis for the recommendation, such as the patient's disease diagnosis being consistent with the trial's target disease, a high degree of fulfillment of inclusion criteria, and a low exclusion risk; when a patient is not recommended, the system can also explain the reasons for rejection, such as disease domain mismatch, potential exclusion risk, or insufficient patient evidence. Compared to black-box models that only output a single score, this invention provides clinical researchers with clearer judgment criteria, helping to improve the efficiency of result review and clinical usability.

[0116] 3. Enhance the ability to diagnose erroneous samples This invention integrates neural re-ranking results and large model verification results into a feedback record, enabling the system to analyze the consistency and discrepancies between the two models. By comparing neural re-ranking scores, large model verification scores, large model verification labels, true labels, and validation metrics, the system can identify different types of errors. For example, if the neural re-ranking model gives a high score, but the true label is negative and the large model indicates a high risk of exclusion, it can be diagnosed as an over-inclusion problem; if the neural re-ranking model gives a low score, but the true label is positive and the large model considers the disease match and inclusion conditions good, it can be diagnosed as an over-exclusion problem; if the model score deviates overall from the label distribution, it can be diagnosed as a calibration problem. This joint diagnostic method based on neural model output and large model verification results can pinpoint the source of errors more accurately than simply looking at the model score.

[0117] 4. Automate and controllable model optimization process This invention employs a multi-agent feedback optimization mechanism, decomposing error diagnosis, update strategy selection, code modification, and result verification into multiple agents. Each agent has a clearly defined responsibility and is subject to rule constraints. The feedback diagnosis agent is responsible for identifying error types; the decision control agent is responsible for selecting the update mode based on error statistics; the execution update agent is responsible for implementing minimal code modifications; and the verification agent is responsible for comparing performance before and after the modifications and deciding whether to adopt the changes. This mechanism reduces the workload of manually analyzing error samples, manually modifying code, and manually comparing results. Compared to manual parameter tuning, this invention improves model iteration efficiency and ensures that each update has clear diagnostic criteria, execution records, and verification results, thereby enhancing the controllability of the optimization process.

[0118] 5. Improve the security and auditability of model updates. This invention backs up the original code during updates, avoiding direct overwriting of the original files. Modified experimental results are output to a new results directory, preventing overwriting of the original results. Simultaneously, the system saves modification prompts, execution logs, standard output, standard error, and runtime summaries for easy auditing. When deciding whether to adopt a modification, the validation agent compares the common metrics of the original and modified models on the validation set. Updates are only accepted when the modification clearly improves performance; rollbacks occur when performance significantly decreases; and manual review is initiated when results are unclear, with automatic adoption of modified results by default. Therefore, this invention avoids the problem of uncontrollable model performance caused by unconstrained automatic modifications, improving the security, traceability, and audit friendliness of the model update process.

[0119] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent patient matching based on multi-stage recommendation clinical trials, characterized in that, include: Obtain patient-trial samples with real-match tags, including subject texts and eligibility criteria texts for both patients and clinical trials; Two types of topic samples are input into the neural reordering model. The topic relevance is ranked, and topic relevance scores are output; the two types of qualification standard texts are input into the neural re-ranking model. The relevance scores of the eligibility criteria are ranked and output as eligibility criteria relevance scores. The re-ranked scores are then combined with the scores of the first two criteria and ranked to screen samples to be reviewed. Here, k is the model version number. The scores and rankings of the samples to be reviewed are used as auxiliary prior information. They are then input into a large language model along with the corresponding patient-trial samples for clinical review. The output is a structured review result that includes review scores, review labels and clinical dimension review information. Based on the true matching labels of the samples to be reviewed, the scores and rankings of each item, and the structured review results, a feedback record is constructed, and the error type and deviation direction are determined by the feedback diagnostic agent to obtain effective diagnostic results. Based on the deviation distribution of effective diagnostic results, the decision control agent selects an update mode of strict matching update, relaxed matching update, parameter modification or no change, and generates an execution specification that limits the updatable configuration items. The first three update modes are used to add predefined clinical input fields and retrain the model, reduce predefined unnecessary clinical input fields and retrain the model, and adjust the score fusion coefficient and / or matching judgment threshold without retraining. The updated neural reordering model is generated by the executing agent based on the execution specification. And compared by the verifying agent and Based on the corresponding evaluation metrics on the same fixed validation set, acceptance is determined. , roll back to Alternatively, manual verification can be performed, and the selected model can output the patient-trial matching score and predicted matching label.

2. The intelligent patient matching method based on multi-stage recommended clinical trials as described in claim 1, characterized in that, The topic relevance score, qualification criterion relevance score, re-ranking score, and ranking of the sample to be reviewed are converted into auxiliary prior information in natural language form, and the auxiliary prior information is written into the prompt information of the input large language model; The prompt information includes a verification instruction that instructs the large language model to use the patient-trial sample as the primary basis for judgment and the auxiliary prior information as supplementary evidence.

3. The intelligent patient matching method based on multi-stage recommended clinical trials as described in claim 1, characterized in that, The clinical dimension review information includes disease matching degree, inclusion criterion satisfaction degree, and exclusion risk degree, and the structured review results are output as parsable structured data objects; The disease matching degree represents the degree of association between the patient's primary diagnosis and the target disease of the clinical trial; the inclusion criterion satisfaction degree represents the degree to which the patient's clinical information meets the clinical trial inclusion criteria; and the exclusion risk degree represents the degree of risk that the patient will meet the clinical trial exclusion criteria.

4. The intelligent patient matching method based on multi-stage recommended clinical trials as described in claim 1, characterized in that, The step of determining the error type and deviation direction by the feedback diagnostic agent to obtain effective diagnostic results includes: When the true matching label is a negative example and meets any of the following conditions: the re-ranking score is not lower than the first high score threshold, or the re-ranking score is not lower than the second high score threshold and the review label is a negative example, the error type is determined to be over-inclusion error and the deviation direction is enhanced matching constraint. When the actual matching label is a positive example and meets any of the following conditions: the re-ranking score is not higher than the first low score threshold, or the re-ranking score is not higher than the second low score threshold and the review label is a positive example, the error type is determined to be an over-exclusion error and the deviation direction is relaxed matching constraints. When the re-ranking score is within the preset boundary range, and the absolute value of the difference between the review score and the re-ranking score reaches the preset difference, the error type is determined to be calibration error and the deviation direction is parameter calibration. If the aforementioned conditions are not met, the error type is determined to be no error and the deviation direction is determined to be no update required; Generate a diagnostic result that includes the error type, the deviation direction, and the diagnostic confidence level; Diagnostic results with a confidence level lower than a preset confidence threshold are filtered out, and the remaining diagnostic results are taken as valid diagnostic results. Wherein, positive examples indicate that the patient is a match for the clinical trial, negative examples indicate that the patient is not a match for the clinical trial, the first high score threshold is greater than the second high score threshold, and the first low score threshold is less than the second low score threshold.

5. The intelligent patient matching method based on multi-stage recommended clinical trials as described in claim 4, characterized in that, The decision-control agent selects an update mode—strict matching update, relaxed matching update, parameter modification, or unchanged update—based on the deviation distribution of effective diagnostic results, including: The number of valid diagnostic results corresponding to overinclusion error, overexclusion error and calibration error, the proportion of each of these numbers to all valid diagnostic results and the corresponding average diagnostic confidence were statistically analyzed. When the number of valid diagnostic results corresponding to calibration errors, their proportion of all valid diagnostic results, and the average diagnostic confidence, as well as the ratio of the larger of the number of valid diagnostic results corresponding to calibration errors to the number of valid diagnostic results corresponding to over-inclusion errors and over-exclusion errors, reach the corresponding preset threshold, it is determined that there is a calibration deviation and the parameter is modified. When the triggering conditions for parameter modification are not met, if the number of valid diagnostic results corresponding to the over-exclusion error reaches the preset minimum number, is greater than the number of valid diagnostic results corresponding to the over-inclusion error, and the preset field number lower limit is not triggered, the relaxed matching update is selected. When the triggering conditions for parameter modification are not met, if the number of valid diagnostic results corresponding to over-inclusion errors reaches the preset minimum number and is greater than the number of valid diagnostic results corresponding to over-exclusion errors, the strict matching update mode is selected. If none of the aforementioned update mode trigger conditions are met, the option is to remain unchanged.

6. The intelligent patient matching method based on multi-stage recommended clinical trials as described in claim 1, characterized in that, The updated neural reordering model is generated by the execution updating agent based on the execution specification. ,include: Backup neural reordering model when the selected update mode is not immutable. The corresponding benchmark programs and benchmark models are generated and stored separately as candidate programs and candidate models; Only modify the input fields, fusion coefficients, matching thresholds and / or result saving paths specified in the execution specifications, while keeping the training process, evaluation logic and irrelevant data streams unchanged; Under strict matching or relaxed matching updates, retrain the candidate model according to the modified input fields; under parameter modifications, maintain the neural reordering model. The network parameters remain unchanged, and the score fusion coefficient and / or matching judgment threshold are adjusted. Running the candidate program and the candidate model yields the updated neural reordering model. And its evaluation results.

7. The intelligent patient matching method based on multi-stage recommended clinical trials as described in claim 1, characterized in that, The comparison by the verification agent and The corresponding evaluation metrics on the same fixed validation set include: Sure and The evaluation results contain common evaluation indicators, which are used as public evaluation indicators. Calculate the absolute and relative changes of each public evaluation indicator according to the pre-defined performance evaluation criteria; if When the corresponding public evaluation indicator value is zero, only the absolute change of that public evaluation indicator is calculated; When the public assessment indicators that meet the first preset ratio improve and the degradation of the remaining public assessment indicators does not exceed the first permissible range, acceptance is granted. ; When a public assessment indicator that has reached the second preset proportion degrades and the degree of degradation exceeds the second allowable range, it will revert to [a previous setting]. ; In all other cases, manual verification is performed, and the evaluation results of both models are retained until the manual verification is completed, with the neural reordering model being used. As the default selected model.

8. A patient intelligent matching system based on multi-stage recommendation clinical trials, characterized in that, include: The sample acquisition module is used to acquire patient-trial samples with real-match labels, including the subject text and eligibility criteria text of each patient and clinical trial. The reordering module is used to input two types of topic samples into the neural reordering model. The topic relevance is ranked, and topic relevance scores are output; the two types of qualification standard texts are input into the neural re-ranking model. The relevance scores of the eligibility criteria are ranked and output as eligibility criteria relevance scores. The re-ranked scores are then combined with the scores of the first two criteria and ranked to screen samples to be reviewed. Here, k is the model version number. The large language model review module is used to take the scores and rankings of the samples to be reviewed as auxiliary prior information, and input them along with the corresponding patient-trial samples into the large language model for clinical review. The output is a structured review result including review score, review label and clinical dimension review information. The feedback record construction and feedback diagnosis module is used to construct feedback records based on the true matching labels of the sample to be reviewed, the scores and rankings of each item, and the structured review results. The feedback diagnosis agent determines the error type and deviation direction to obtain effective diagnostic results. The decision control module is used by the decision control agent to select an update mode of strict matching update, relaxed matching update, parameter modification or no change based on the deviation distribution of effective diagnostic results, and generate an execution specification that limits the updatable configuration items. The first three update modes are used to add predefined clinical input fields and retrain the model, reduce predefined unnecessary clinical input fields and retrain the model, and adjust the score fusion coefficient and / or matching judgment threshold without retraining. The execution update and verification module is used by the execution update agent to generate the updated neural reordering model based on the execution specification. And compared by the verifying agent and Based on the corresponding evaluation metrics on the same fixed validation set, acceptance is determined. , roll back to Alternatively, manual verification can be performed, and the selected model can output the patient-trial matching score and predicted matching label.

9. A storage medium, characterized in that, It stores a computer program, wherein the computer program causes the computer to execute the intelligent patient matching method based on multi-stage recommended clinical trials as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes: A processor and a memory; the memory stores program instructions; the processor is configured to run the program instructions to execute the intelligent patient matching method based on multi-stage recommendation clinical trials as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Intelligent mining method and system for potential subjects in clinical research based on cooperation of large and small models

    CN121862284A