Medical auxiliary decision evaluation method and device based on real world longitudinal data, and electronic equipment

Abstract

The present disclosure relates to a medical auxiliary decision evaluation method and device based on real-world longitudinal data, and an electronic device. The method comprises: determining health data of a sample patient at each decision point to obtain a health data time sequence, wherein the health data comprises health status data, real medical operation data, and medical outcome data; determining historical health data of a to-be-predicted decision point based on the health data time sequence; determining model to-be-processed data; inputting the model to-be-processed data into a candidate model to obtain predicted medical operation data; and processing the predicted medical operation data, the real medical operation data, covariate information, and the medical outcome data of the sample patient at a plurality of target decision points based on time-varying causal inference to obtain a value estimation result. The method can improve the accuracy, objectivity, and effectiveness of the evaluation of the auxiliary medical decision-making capability of the candidate model. Moreover, the method does not involve actual ethical issues, thereby reducing the evaluation time and cost.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a medical auxiliary decision-making evaluation method, device, and electronic device based on real-world longitudinal data. Background Technology

[0002] With the development of generative artificial intelligence, multimodal medical models, and clinical decision support systems, more and more models are being used to recommend personalized medical procedures such as medication, examinations, surgeries, rehabilitation, and follow-ups.

[0003] However, when evaluating such models, the evaluation is often based on static question-answer pairs in a benchmark set, or on local consistency comparisons at a single point in time.

[0004] For example, a single model recommendation might be matched with a medical order or a piece of text based on similarity, and the degree of matching is used as the model's evaluation result. This type of evaluation primarily reflects the consistency between the model's recommendations and actual medical practices or clinical guidelines, and is unlikely to indicate the clinical effectiveness of the recommendations.

[0005] Furthermore, evaluations based solely on a single recommendation cannot comprehensively consider changes in a patient's health status over time, or the cumulative impact of medical procedures at multiple decision points on patient outcomes. This makes it difficult to accurately evaluate candidate models from the perspective of the impact of dynamic medical decisions formed by the candidate model on long-term clinical outcomes, even under the assumption that treatment is consistently implemented according to the candidate model's recommendations at multiple decision points in the patient's care process.

[0006] Conducting controlled trials would involve long trial periods, high costs, and ethical issues. Summary of the Invention

[0007] In view of this, this disclosure proposes a medical decision support evaluation scheme based on real-world longitudinal data.

[0008] According to one aspect of this disclosure, a method for evaluating medical assistance decision-making based on real-world longitudinal data is provided. The method includes: determining health data of sample patients at various decision-making points to obtain a time series of health data for the sample patients, the health data including: health status data, actual medical operation data, and medical outcome data; determining historical health data corresponding to the decision-making point to be predicted based on the health data time series; determining model data to be processed, the model data to be processed including the historical health data corresponding to the decision-making point to be predicted; inputting the model data to be processed into a candidate model to obtain predicted medical operation data corresponding to the decision-making point to be predicted; and processing the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data corresponding to multiple sample patients at multiple target decision-making points based on a time-varying causal inference algorithm to obtain a value estimation result for evaluating the value of the medical assistance decision-making strategy of the candidate model, wherein the covariate information characterizes the baseline and time-varying conditions of the sample patients.

[0009] In one possible implementation, based on a time-varying causal inference algorithm, the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data corresponding to multiple sample patients at multiple target decision points are processed to obtain a value estimation result for evaluating the value of the auxiliary medical decision-making strategy of the candidate model. This includes: for each sample patient, determining a data similarity index between the actual medical operation data and the predicted medical operation data corresponding to the target decision point; processing the data similarity index, covariate information, and medical outcome data of multiple sample patients at the multiple target decision points based on an inverse probability weighting algorithm to obtain a first value estimation result; or, processing the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of multiple sample patients at the multiple target decision points based on a G-Formula algorithm to obtain a second value estimation result; or, processing the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of each sample patient at the multiple target decision points based on a combined algorithm of the inverse probability weighting algorithm and the G-Formula algorithm to obtain a third value estimation result.

[0010] In one possible implementation, for each sample patient, determining the data similarity index between the real medical operation data and the predicted medical operation data corresponding to the target decision point includes: determining the comparison dimension of the real medical operation data and the predicted medical operation data at the target decision point based on the type of medical operation, wherein the comparison dimension is determined by fields in the medical operation data; determining the dimension similarity index of the real medical operation data and the predicted medical operation data for each comparison dimension; and determining the data similarity index based on the dimension similarity index.

[0011] In one possible implementation, determining the health data of the sample patient at each decision point includes: determining multiple decision points based on the patient's disease course; acquiring the overall health data of the sample patient, wherein each sub-item of the overall health data includes a timestamp; identifying abnormal data in the overall health data; correcting the abnormal data using different strategies based on the abnormality type of the abnormal data to obtain optimized overall health data; and determining the health data corresponding to each decision point based on the alignment of timestamps and decision points for the optimized overall health data.

[0012] In one possible implementation, the anomaly types include: multiple timestamps for the same event, timestamp conflicts, missing timestamps, and missing data. Based on the anomaly types, different strategies are used to correct the anomaly data to obtain optimized overall health data. This includes: when the anomaly data consists of multiple timestamps for the same event, determining a target timestamp from multiple candidate timestamps according to preset rules based on the event to which the anomaly data belongs; when the anomaly data consists of timestamp conflicts, correcting the conflicting timestamps based on adjacent events and / or the hospitalization stage to which the anomaly data belongs; and when the anomaly data consists of missing timestamps or missing data, supplementing the missing timestamps or missing data based on adjacent events and / or the hospitalization stage to which the anomaly data belongs.

[0013] In one possible implementation, determining the model data to be processed includes: using historical health data corresponding to the decision point to be predicted as the model data to be processed; or using real medical operation data and historical health data corresponding to the decision point to be predicted as the model data to be processed.

[0014] In one possible implementation, the step of inputting the data to be processed by the model into the candidate model to obtain the predicted medical operation data corresponding to the decision point to be predicted includes: inputting the historical health data corresponding to the decision point to be predicted into the candidate model; and inputting prompt words into the candidate model so that the candidate model obtains the predicted medical operation data corresponding to the decision point to be predicted based on the historical health data of the decision point to be predicted.

[0015] In one possible implementation, the health status data at least represents: physical indicator values, medical event records, source system, reference interval, anomaly markers, evidence locations, missing markers, and timestamps corresponding to the physical indicator values ​​and the medical event records; the real medical operation data includes one or more of the following: medical operation name, standard terminology code for the medical operation, medical operation timestamp, medical operation type, target or site of action, dosage or intensity, frequency, route, and course of treatment; the medical outcome data includes at least one of the following: disease progression, complications, in-hospital adverse events, discharge method, readmission, death, functional recovery, and medical costs.

[0016] According to another aspect of this disclosure, a medical assistance decision-making assessment device based on real-world longitudinal data is provided, comprising:

[0017] A health data time series determination unit is used to determine the health data of the sample patients at each decision point, thereby obtaining the health data time series of the sample patients. The health data includes: health status data, actual medical operation data, and medical outcome data.

[0018] The historical health data determination unit is used to determine the historical health data corresponding to the decision point to be predicted based on the health data time series.

[0019] The model data to be processed determination unit is used to determine the model data to be processed, which includes historical health data corresponding to the decision point to be predicted.

[0020] The predictive medical operation data determination unit is used to input the data to be processed in the model into the candidate model to obtain the predictive medical operation data corresponding to the decision point to be predicted.

[0021] The value estimation result determination unit is used to process the predicted medical operation data, real medical operation data, covariate information, and medical outcome data of multiple sample patients at multiple target decision time points based on a time-varying causal inference algorithm, so as to obtain the value estimation result for evaluating the value of the auxiliary medical decision-making strategy of the candidate model. The covariate information represents the baseline and time-varying conditions of the sample patients.

[0022] In one possible implementation, the value estimation result determination unit is further configured to:

[0023] For each sample of patients, a data similarity index is determined between the actual medical operation data and the predicted medical operation data corresponding to the target decision point;

[0024] Based on the inverse probability weighted algorithm, the data similarity indicators, covariate information, and medical outcome data of multiple sample patients at the multiple target decision time points are processed to obtain the first value estimation result;

[0025] or,

[0026] Based on the G-Formula algorithm, the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of multiple sample patients at multiple target decision points are processed to obtain a second value estimation result.

[0027] or,

[0028] Based on a combination of the inverse probability weighted algorithm and the G-Formula algorithm, the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of each sample patient at the multiple target decision points are processed to obtain a third value estimation result.

[0029] In one possible implementation, the value estimation result determination unit is further configured to:

[0030] Based on the type of medical operation, a comparison dimension is determined between the actual medical operation data and the predicted medical operation data at the target decision point, and the comparison dimension is determined by the fields in the medical operation data.

[0031] Determine the dimensional similarity indicators between real medical operation data and predicted medical operation data in each comparison dimension;

[0032] The data similarity indicators are determined based on the similarity indicators of each dimension.

[0033] In one possible implementation, the health data time series determination unit is further configured to:

[0034] Based on the disease course of the sample patients, multiple decision-making time points were determined;

[0035] Obtain the overall health data of the sample patients, wherein each sub-item of the overall health data includes a timestamp;

[0036] Identify abnormal data in overall health data;

[0037] Based on the anomaly type of the abnormal data, different strategies are used to correct the abnormal data and obtain optimized overall health data.

[0038] To optimize overall health data, health data corresponding to each decision point is determined based on timestamps and decision point alignment.

[0039] In one possible implementation, the exception types include: multiple timestamps for the same event, timestamp conflicts, missing timestamps, and missing data;

[0040] The health data time series determination unit is further used for:

[0041] In the case where the abnormal data consists of multiple timestamps of the same event, the target timestamp is determined from multiple candidate timestamps based on the event to which the abnormal data belongs, according to a preset rule.

[0042] In the case of timestamp conflicts in the abnormal data, the conflicting timestamps are corrected based on the adjacent events and / or the hospitalization stage to which the abnormal data belongs;

[0043] In the case where the abnormal data is missing timestamps or missing data, the missing timestamps or missing data are supplemented based on the events adjacent to the abnormal data and / or the hospitalization stage to which they belong.

[0044] In one possible implementation, the model data determination unit is further used for:

[0045] Historical health data corresponding to the decision-making time point to be predicted is used as the data to be processed by the model.

[0046] or,

[0047] The actual medical operation data and historical health data corresponding to the decision-making time point to be predicted are used as the data to be processed by the model.

[0048] In one possible implementation, the predictive medical operation data determination unit is further configured to:

[0049] Input the historical health data corresponding to the decision point to be predicted into the candidate model;

[0050] The prompt words are input into the candidate model so that the candidate model can obtain the predicted medical operation data corresponding to the decision point to be predicted based on the historical health data of the decision point to be predicted.

[0051] In one possible implementation, the health status data at least represents: physical indicator values, medical event records, source system, reference interval, anomaly markers, evidence locations, missing markers, and timestamps corresponding to the physical indicator values ​​and the medical event records;

[0052] The actual medical operation data includes one or more of the following: medical operation name, standard terminology code of medical operation, medical operation timestamp, medical operation type, target or site of action, dosage or intensity, frequency, route, and course of treatment;

[0053] Medical outcome data should include at least one of the following: disease progression, complications, in-hospital adverse events, discharge mode, readmission, death, functional recovery, and medical costs.

[0054] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0055] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0056] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0057] In this embodiment, based on the time series of health data of sample patients, predictive medical action data corresponding to each decision point is determined. Instead of considering only the predictive ability of a single time point, the predictive medical action data from multiple decision points are used as the evaluation object. Based on the longitudinal temporal relationship between medical action and medical outcome, a time-varying causal inference algorithm is used to estimate the medical outcome corresponding to the continued adoption of the recommendation results output by the candidate model, thereby evaluating the candidate model's ability to "develop personalized strategies that improve patients' long-term medical outcomes." Furthermore, by using covariate information as boundary constraints, the value estimation results can consider the actual health status of different patients at different time points, improving the evaluation power. Therefore, the method of this disclosure can improve the accuracy, objectivity, and power of evaluating the value of candidate model-assisted medical decision-making strategies. Moreover, it avoids practical ethical issues and reduces evaluation time and cost.

[0058] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0059] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0060] Figure 1 A flowchart illustrating the medical decision-making evaluation method based on real-world longitudinal data provided in this embodiment of the disclosure.

[0061] Figure 2 This is a schematic diagram illustrating the comparison of multiple outcome effects of patient samples from Hospital A following a candidate model and medical operation data provided by human doctors in scenario 1, as provided in an embodiment of this disclosure.

[0062] Figure 3 This is a schematic diagram comparing the probability of no-event outcomes in bleeding outcomes for a sample of patients in Hospital A following a candidate model and medical operation data provided by human doctors, as provided in an embodiment of this disclosure in Scenario 1.

[0063] Figure 4 This is a schematic diagram showing the comparison of the probability of no events in major adverse cardiovascular events for a sample of patients in Hospital A following a candidate model and medical operation data provided by human doctors, as provided in an embodiment of this disclosure in Scenario 1.

[0064] Figure 5 This is a schematic diagram illustrating the comparison of multiple outcome effects of sample patients following candidate models and medical operation data provided by human doctors in the intensive care medical database under scenario 1, as provided in an embodiment of this disclosure.

[0065] Figure 6 This is a schematic diagram comparing the probability of no-event outcomes in bleeding outcomes for sample patients in the intensive care medical database following the candidate model and the medical operation data provided by human doctors in Scenario 1, as provided in this embodiment of the disclosure.

[0066] Figure 7 This is a schematic diagram showing the comparison of the probability of no major adverse cardiovascular events in the sample patients in the intensive care medical database following the candidate model and the medical operation data provided by human doctors in Scenario 1, as provided in the embodiments of this disclosure.

[0067] Figure 8 A schematic diagram of the structure of a medical auxiliary decision-making evaluation system based on real-world longitudinal data provided in an embodiment of this disclosure.

[0068] Figure 9 A flowchart illustrating the process of determining data similarity indicators provided in this embodiment of the disclosure.

[0069] Figure 10 This is a schematic flowchart illustrating the evaluation method for candidate models provided in an embodiment of this disclosure.

[0070] Figure 11 A schematic diagram of the structure of a medical auxiliary decision-making and evaluation device based on real-world longitudinal data provided in an embodiment of this disclosure.

[0071] Figure 12 A schematic diagram of the structure of an electronic device for medical decision-making evaluation based on real-world longitudinal data, provided in an embodiment of this disclosure. Detailed Implementation

[0072] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0073] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0074] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0075] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0076] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0077] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0078] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0079] Figure 1 This is a flowchart illustrating a medical decision support and evaluation method based on real-world longitudinal data, provided in an embodiment of this disclosure. Figure 1 As shown, the method includes:

[0080] S11, determine the health data of the sample patients at each decision point, and obtain the health data time series of the sample patients. The health data includes: health status data, actual medical operation data, and medical outcome data. The health data time series includes: health status time series, actual medical operation time series, and medical outcome time series.

[0081] The sample patients can be confirmed patients, and / or suspected patients, and / or follow-up patients in the hospital. The sample patients can also be patients recorded in a medical database. Health data can characterize the entire or partial patient status and treatment measures taken during the process from illness to stabilization after medical treatment.

[0082] Health data includes: health status data, actual medical procedure data, and medical outcome data. Health status data at least represents: physical indicator values, medical event records, source system, reference intervals, anomaly markers, evidence locations, missing data markers, and timestamps corresponding to the physical indicator values ​​and medical event records. Actual medical procedure data may include one or more of the following: medical procedure name, standard terminology code for the medical procedure, medical procedure timestamp, medical procedure type, target or site of action, dosage or intensity, frequency, route, and treatment course to which the medical procedure belongs. Medical outcome data at least includes one of the following: disease progression, complications, in-hospital adverse events, discharge method, readmission, death, functional recovery, and medical costs. Medical outcome data includes not only patient treatment outcomes and effects but also medical costs, allowing the value estimation results obtained in subsequent evaluations of candidate model-assisted medical decision-making strategies to include multiple model evaluation dimensions. Model evaluation dimensions may include: treatment outcomes, and / or treatment effects, and / or treatment costs, etc.

[0083] A decision point in time can be defined as a specific point in a patient's medical history where a medical procedure is required. The corresponding health data for each decision point can be the health data generated between the previous decision point and the current decision point. Arranging the health data for each decision point in chronological order yields a health data time series. Health data includes health status data, actual medical procedure data, and medical outcome data. Therefore, a health data time series can include: a health status time series, an actual medical procedure time series, and a medical outcome time series.

[0084] S12, Based on the health data time series, determine the historical health data corresponding to the decision point to be predicted.

[0085] The decision point to be predicted can be any decision point.

[0086] The historical health data corresponding to the decision point to be predicted can be the collection of health data from all previous decision points. The health data corresponding to each decision point can be clearly identified from the health data time series; therefore, the historical health data corresponding to the decision point to be predicted can be identified from the health data time series. Historical health data includes historical health status data, historical actual medical operation data, and historical medical outcome data.

[0087] S13, determine the model data to be processed, the model data to be processed includes historical health data corresponding to the decision point to be predicted.

[0088] For example, historical health data corresponding to the decision point to be predicted can be used as the data to be processed by the model. This allows candidate models to directly recommend medical procedures for the decision point to be predicted based on historical health status data, historical actual medical operation data, and historical medical outcome data, facilitating a more objective evaluation of the candidate models' independent medical decision-making capabilities.

[0089] For example, real-world medical procedure data and historical health data corresponding to the decision-making point in time can be used as the data to be processed by the model. In this example, based on the decision-making point in time, the real-world medical procedure data and historical health data (i.e., historical health status data, historical real-world medical procedure data, and historical medical outcome data) corresponding to the decision-making point in time can be used as the data to be processed by the model. In this way, the candidate model can provide suggestions for correction based on historical health data and real-world medical procedure data at the decision-making point in time, which facilitates an objective evaluation of the candidate model's ability to assist in medical decision-making.

[0090] S14, input the data to be processed in the model into the candidate model to obtain the predicted medical operation data corresponding to the decision point to be predicted.

[0091] This disclosure does not limit the type or number of candidate models. Candidate models can be medical artificial intelligence models. Candidate models can output recommended medical operations corresponding to the predicted decision point based on the data to be processed by the model. For ease of description, the medical operations output by the candidate model are named predicted medical operation data. Since the predicted decision point can be any decision point, the candidate model can output predicted medical operation data corresponding to any decision point.

[0092] S15, based on the time-varying causal inference algorithm, process the predicted medical operation data, real medical operation data, covariate information, and medical outcome data corresponding to multiple sample patients at multiple target decision time points, so as to obtain the value estimation result for evaluating the value of the auxiliary medical decision-making strategy of the candidate model. The covariate information represents the baseline and time-varying conditions of the sample patients.

[0093] Time-varying causal inference algorithms are time-series data analysis methods specifically designed to identify and estimate the causal effects of time-varying interventions on outcomes. In medical scenarios, a longitudinal temporal relationship exists between medical procedures and medical outcomes. By using the sequence of medical procedures at each target decision point as intervention variables and the medical outcomes within a pre-defined follow-up period as outcome variables, the impact of different medical procedure strategies on medical outcomes can be estimated while satisfying pre-defined causal identification assumptions. Therefore, time-varying causal inference algorithms can effectively evaluate the decision-making support value of candidate models in real-world medical decision-making scenarios.

[0094] Covariate information characterizes the baseline and time-varying conditions of the sample patients. Baseline conditions may include the sample patients' baseline health level and demographic characteristics. Time-varying conditions may include the sample patients' health level and medical procedures at each decision point. Data reflecting the baseline health level may include: patient medical history, baseline health status, comorbidity information, previous treatment history, and health status data. Demographic characteristics can be extracted from the demographic data. Therefore, covariate information may include one or more of the following data: patient medical history, demographic data, baseline health status, comorbidity information, previous treatment history, and time-series health status characteristics.

[0095] In this embodiment, the value estimation results can be used as the basis for ranking multiple candidate models. The value estimation results include: the expected outcome value of the medical assistance decision-making strategy corresponding to the candidate model on the preset medical outcome; the estimated value corresponding to the candidate model. Here, the estimated value represents the risk difference, risk ratio, or average strategy effect of the medical assistance decision-making strategy relative to the preset reference strategy; and the confidence interval and significance test results corresponding to the aforementioned estimated value. Based on the value estimation results of multiple candidate models, the multiple candidate models are ranked to obtain a ranking result. The ranking result includes the ranking rank and / or ranking score of each candidate model, and at least one of the following: the probability of becoming the optimal candidate model, the sub-outcome ranking results for different preset medical outcomes, or the comprehensive utility ranking result determined based on a preset utility function.

[0096] In this embodiment of the disclosure, a decision point for evaluation can be selected. For ease of description, the selected decision point for evaluation is named the target decision point. Using the target decision point, predicted medical operation data and actual medical operation data corresponding to different time points are filtered from the time series of health data of the sample patients.

[0097] Based on the time-varying causal inference algorithm, the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of multiple sample patients at the target decision time point are processed to obtain the value estimation results of the candidate model.

[0098] In this embodiment, based on the time series of health data of sample patients, predictive medical action data corresponding to each decision point is determined. Instead of considering only the predictive ability of a single time point, the predictive medical action data from multiple decision points are used as the evaluation object. Based on the longitudinal temporal relationship between medical action and medical outcome, a time-varying causal inference algorithm is used to estimate the medical outcome corresponding to the continued adoption of the recommendation results output by the candidate model, thereby evaluating the candidate model's ability to "develop personalized strategies that improve patients' long-term medical outcomes." Furthermore, by using covariate information as boundary constraints, the value estimation results can consider the actual health status of different patients at different time points, improving the evaluation power. Therefore, the method of this disclosure can improve the accuracy, objectivity, and power of evaluating the value of candidate model-assisted medical decision-making strategies. Moreover, it avoids practical ethical issues and reduces evaluation time and cost.

[0099] In one possible implementation, based on a time-varying causal inference algorithm, the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data corresponding to multiple sample patients at multiple target decision points are processed to obtain a value estimation result for evaluating the value of the auxiliary medical decision-making strategy of the candidate model. This includes: for each sample patient, determining a data similarity index between the actual medical operation data and the predicted medical operation data corresponding to the target decision point; processing the data similarity index, covariate information, and medical outcome data of multiple sample patients at the multiple target decision points based on an inverse probability weighting algorithm to obtain a first value estimation result; or, processing the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of multiple sample patients at the multiple target decision points based on a G-Formula algorithm to obtain a second value estimation result; or, processing the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of each sample patient at the multiple target decision points based on a combined algorithm of the inverse probability weighting algorithm and the G-Formula algorithm to obtain a third value estimation result.

[0100] This disclosure provides three time-varying causal inference algorithms: Method 1, an inverse probability weighted algorithm; Method 2, a G-Formula algorithm; and Method 3, a combination of the inverse probability weighted algorithm and the G-Formula algorithm. Any of these three methods can be used to evaluate the auxiliary medical decision-making ability of candidate models. Methods 1 and 2 are single-robust estimation algorithms, while Method 3 is a double-robust estimation algorithm.

[0101] When both predicted and actual medical operation data can be encoded into structured vectors, for Method 2 or Method 3, the structured predicted and actual medical operation data, along with covariates and medical outcome data, can be used as input data to obtain value estimation results.

[0102] In cases where it is impossible to encode both predicted and actual medical operation data into structured vectors, a first representation vector for representing actual medical operation data and a second representation vector for representing predicted medical operation data can be constructed based on the data similarity index. The first and second representation vectors are used to represent the relative consistency between actual and predicted medical operation data at various target decision points. Furthermore, for method two or method three, the first and second representation vectors, covariate information, and medical outcome data can be used as input data to obtain the value estimation result.

[0103] Method 1 does not directly use real medical operation data and predicted medical operation data. Instead, it obtains a first value estimation result based on data similarity indicators, covariates, and medical outcome data. Specifically, when the data similarity indicator is a binary variable, it determines whether the current real medical operation conforms to the recommended medical operation output by the candidate model. Sample-time records where the current real medical operation does not conform to the recommended medical operation output by the candidate model are manually censored. The censoring mechanism is modeled to calculate the inverse probability censoring weight, and then the first value estimation result is calculated. This binary variable can represent whether the current real medical operation conforms to or does not conform to the recommended medical operation output by the candidate model.

[0104] When the data similarity index is a continuous variable, it can be converted into a binary consistency index based on a preset threshold. Based on the binary consistency index, the steps described above—from determining whether the current actual medical operation matches the recommended medical operation output by the candidate model to calculating the first value estimation result—can be performed. Alternatively, the continuous data similarity index can be used as a time-varying continuous exposure variable, and the inverse probability density weight can be calculated using a method based on conditional density ratio to calculate the first value estimation result. If there are actual loss to follow-up, interrupted follow-up, or administrative censoring, a loss to follow-up censoring weight can be further calculated, and the loss to follow-up censoring weight can be multiplied by the inverse probability censoring weight or the inverse probability density weight to obtain the total weight. The total weight is then used to determine the first value estimation result.

[0105] Data similarity metrics characterize the degree of similarity between real and predicted medical procedure data. For numerical data, data similarity metrics can be the absolute difference, relative difference, or normalized difference between recommended and actual numerical data. For categorical data, similarity metrics can be determined based on field consistency, medical terminology coding consistency, hierarchical distance of medical terminology coding, or category mapping relationships. For descriptive data, data similarity metrics can be the semantic similarity between recommended and actual descriptive data. Categorical data refers to data whose values ​​belong to a predefined set of categories; descriptive data refers to data that uses natural language to describe medical procedure content. Text length can be used as an auxiliary condition for determining data type, but it is not the sole basis for distinguishing between categorical and descriptive data.

[0106] The above are merely examples, and this disclosure does not limit the data similarity indicators. Another method for determining data similarity indicators will be described below.

[0107] In this embodiment of the disclosure, in order to improve the applicability of the method to different types of medical operation data, field matching, dimensional similarity calculation, field combination matching, medical entity recognition, medical terminology mapping, semantic similarity calculation and / or manual annotation rules can be used to determine the data similarity index between real medical operation data and predicted medical operation data for structured medical operation data, semi-structured medical operation data or unstructured medical operation text, respectively.

[0108] In one possible implementation, for each sample patient, determining the data similarity index between the real medical operation data and the predicted medical operation data corresponding to the target decision point includes: determining the comparison dimension of the real medical operation data and the predicted medical operation data at the target decision point based on the type of medical operation, wherein the comparison dimension is determined by fields in the medical operation data; determining the dimension similarity index of the real medical operation data and the predicted medical operation data for each comparison dimension; and determining the data similarity index based on the dimension similarity index.

[0109] The dimensions for comparing real and predicted medical procedure data can be determined based on the type of medical procedure being compared. For medication-related procedures, fields that can be used as comparison dimensions include: drug name, drug specifications, dosage, frequency of administration, route of administration, and duration. For surgical procedures, fields that can be used as comparison dimensions include: procedure name, surgical site, surgical approach, surgical scope, and time of surgical execution. For examination-related procedures, fields that can be used as comparison dimensions include: examination name, examination site, examination purpose, and time of examination execution. For rehabilitation, follow-up, or nursing procedures, the procedure name, frequency, duration, execution time, and / or objective can be used as comparison dimensions.

[0110] Alternatively, multiple fields can be combined based on an event to obtain a field combination. This field combination can then be used as a comparison dimension. For example, for a drug administration event, the three comparison dimensions of drug type, dosage, and route of administration can be combined.

[0111] For example, dimensional similarity indicators are determined for each comparison dimension. Dimensional similarity indicators characterize the degree of similarity between corresponding real and predicted medical operation data within the same comparison dimension. The dimensional similarity indicators for each comparison dimension are weighted and aggregated to obtain a data similarity indicator between the real and predicted medical operation data. Comparisons are made across all dimensions, allowing the data similarity indicator to more comprehensively reflect the similarity between the real and predicted medical operation data, thus improving the reliability of the data similarity indicator. When the comparison dimensions include field combinations, the comparison dimensions can improve the overall ability to reconstruct events, making the comparison results between the real and predicted medical operation data more reliable.

[0112] For example, for medical operation data that is difficult to match using fields or combinations of fields, medical entity recognition, terminology standardization, and text embedding encoding can be performed on both real and predicted medical operation data. Data similarity indicators can then be determined based on entity matching scores, terminology encoding distance, semantic similarity, and / or cosine similarity of embedding vectors. Thus, even if the medical operation data contains natural language descriptions, non-standardized expressions, or partially missing fields, the degree of similarity between real and predicted medical operation data can be determined.

[0113] Therefore, the method of this disclosure can improve the reliability of similarity indicators and further improve the accuracy and reliability of the evaluation of the auxiliary medical decision-making ability of candidate models.

[0114] In one possible implementation, determining the health data of the sample patient at each decision point includes: determining multiple decision points based on the patient's disease course; acquiring the overall health data of the sample patient, wherein each sub-item of the overall health data includes a timestamp; identifying abnormal data in the overall health data; correcting the abnormal data using different strategies based on the abnormality type of the abnormal data to obtain optimized overall health data; and determining the health data corresponding to each decision point based on the alignment of timestamps and decision points for the optimized overall health data.

[0115] In this embodiment of the disclosure, the course of the disease or sub-courses can be divided according to a fixed duration to determine the decision-making time point. For example, if the preset duration is 12 hours, the first sub-course after the patient enters the intensive care unit can be divided into decision-making time points every 12 hours.

[0116] In this embodiment, the time points of a patient's disease course or sub-courses can be determined as decision points. For example, each hospitalization point can be determined as a decision point; the time points before and after surgery can be determined as decision points; the time points when changes in medical orders occur can be determined as decision points; the time points when complications occur can be determined as decision points; the time points when discharged can be determined as decision points; and the time points for follow-up can be determined as decision points. Multiple decision points can be dynamically generated for different patients or different disease courses of the same patient.

[0117] In this embodiment of the disclosure, overall health data can be determined based on the medical records of sample patients. Medical records can contain various medical data with temporal relationships. Medical records may include one or more of the following record data: outpatient records, inpatient records, examination and testing records, imaging reports, surgical records, anesthesia records, nursing records, prescription records, follow-up records, and medication records.

[0118] Overall health data can be health data at points where no matching decision points exist.

[0119] For ease of description, the data included in the overall health data are named sub-items. Each sub-item can correspond to a timestamp in which it was generated. Therefore, the overall health data includes not only sub-items but also their timestamps.

[0120] Based on pre-trained models or manual methods, abnormal data in the overall health data can be identified, as well as the types of abnormalities to which the abnormal data belongs.

[0121] Based on a strategy that matches the anomaly type to which the anomaly data belongs, the anomaly data is corrected, making the optimized overall data more complete and reasonable after correction. This aims to eliminate or reduce the negative impact of anomaly data on the generation of recommended medical actions, and further improve the evaluation accuracy of candidate models.

[0122] A time interval is defined by two adjacent decision points. Sub-item data whose timestamps fall within this time interval are mapped to the decision point at the end of that time interval. This allows us to obtain the correspondence between each sub-item data point and the decision point within the overall health data. Based on this correspondence, the health data (sub-item data) corresponding to each decision point can be determined.

[0123] In this embodiment, the decision-making time point can be flexibly determined for patients with different diseases or patients with the same disease at different stages. This makes the construction of health data time series more reasonable and closely reflects the actual situation of patients, and also makes the prediction of medical operation data more practical. Furthermore, by selecting matching strategies to correct abnormal data based on the anomaly type, the rationality and completeness of the health data time series are improved. Therefore, this embodiment can improve the rationality of the output results (predicted medical operation data) of the candidate model and reduce the probability of output bias caused by problems with the input data itself. This makes the evaluation of the candidate model more objective and accurate.

[0124] In one possible implementation, the anomaly types include: multiple timestamps for the same event, timestamp conflicts, missing timestamps, and missing data. Based on the anomaly types, different strategies are used to correct the anomaly data to obtain optimized overall health data. This includes: when the anomaly data consists of multiple timestamps for the same event, determining a target timestamp from multiple candidate timestamps according to preset rules based on the event to which the anomaly data belongs; when the anomaly data consists of timestamp conflicts, correcting the conflicting timestamps based on adjacent events and / or the hospitalization stage to which the anomaly data belongs; and when the anomaly data consists of missing timestamps or missing data, supplementing the missing timestamps or missing data based on adjacent events and / or the hospitalization stage to which the anomaly data belongs.

[0125] For anomalous data with multiple timestamps for the same event, a target timestamp can be determined from the multiple timestamps corresponding to the sub-item data based on priority rules, event rules, or time consistency rules. For example, if the sub-item data is a hospital admission event, it corresponds to three timestamps. Two of these timestamps are generated due to system errors, making them predate the patient's visit time, which is illogical. In this case, a target timestamp can be selected based on time consistency rules. Another example: if a laboratory indicator corresponds to two timestamps—one being the time the doctor orders the test, and the other being the time the indicator is generated—the timetamp used to generate the indicator can be selected as the target timestamp based on the event rules for the laboratory test event.

[0126] The above examples are for illustrative purposes only and do not represent all possible scenarios. Other events that use the priority rules, event rules, or time consistency rules in this disclosure to determine the target timestamp are also protected under this disclosure.

[0127] Another scenario involves a sub-item data point corresponding to a timestamp, but this timestamp logically conflicts with or is inverted compared to the timestamps of other sub-items. In other words, the anomalous data falls under the category of timestamp conflicts. In this case, an accurate timestamp can be estimated based on adjacent events to the sub-item data (the anomalous data) and / or the hospitalization stage to which the sub-item data belongs, or the inverted timestamp can be corrected.

[0128] In another scenario, due to asynchronous sampling, omissions, or data transmission issues, there may be missing sub-item data or missing timestamps for the sub-item data. That is, the anomalous data falls into the category of missing timestamps or missing data. In this case, the missing sub-item data or its missing timestamp can be inferred based on adjacent events to the sub-item data (abnormal data) and / or the hospitalization stage to which the sub-item data belongs.

[0129] Using the methods of this disclosure, several types of abnormal data, such as multiple timestamps for the same event, timestamp conflicts, missing timestamps, and missing data, can be corrected, thereby improving and optimizing the quality of overall health data.

[0130] In one possible implementation, the step of inputting the data to be processed by the model into the candidate model to obtain the predicted medical operation data corresponding to the decision point to be predicted includes: inputting the historical health data corresponding to the decision point to be predicted into the candidate model; and inputting prompt words into the candidate model so that the candidate model obtains the predicted medical operation data corresponding to the decision point to be predicted based on the historical health data of the decision point to be predicted.

[0131] In this embodiment, the candidate model can be a large language model. The decision point to be predicted can be any decision point. The prompt words inform the large language model to output recommended medical operations based on historical health data, or to modify the current actual medical operation data based on historical health data to generate predicted medical operation data.

[0132] As mentioned earlier, the data to be processed by the model can be historical health data corresponding to at least one decision point in time. Therefore, the prompt word and the historical health data corresponding to the decision point to be predicted can be input into the candidate model to obtain the predicted medical operation data corresponding to that decision point.

[0133] This makes the recommendation results of the candidate models more in line with human reading habits and facilitates an objective evaluation of the candidate models' ability to assist in medical decision-making when reference data is available.

[0134] In one example, overall health data of sample patients can be obtained. Abnormal data and abnormality types are identified from the overall health data, and a strategy is selected based on the abnormality type to correct the abnormal data, resulting in optimized overall health data. Furthermore, based on the disease course of the sample patients, multiple decision points are determined. A time interval is formed by two adjacent decision points. Sub-items in the optimized overall health data whose timestamps fall within the time interval are mapped to the decision point at the end of that time interval, thus obtaining the correspondence between each sub-item in the overall health data and the decision point. Based on this correspondence, the health data (sub-item data) corresponding to each decision point can be determined. Therefore, the time series of the sample patients' health data can be obtained.

[0135] Based on the time series of health data of sample patients, the historical health data corresponding to the decision point to be predicted is determined. Historical health data may include: historical health status data, historical actual medical operation data, and historical medical outcome data. The historical health data of the decision point to be predicted can be used as the model's processing data, or both the historical health data and actual medical operation data of the decision point to be predicted can be used as the model's processing data.

[0136] Input the data to be processed into the candidate model, and the candidate model can output the predicted medical operation data corresponding to the decision point to be predicted.

[0137] By using any one of the three time-varying causal inference algorithms provided in this disclosure to process the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data corresponding to multiple sample patients, a value estimation result for the candidate model's auxiliary medical decision-making strategy can be obtained. If multiple candidate models exist, each candidate model can be evaluated separately to obtain the corresponding value estimation result for each candidate model. Based on the value estimation results, the target candidate model with the strongest auxiliary medical decision-making ability and the ability to achieve better long-term outcomes is selected.

[0138] Scenario 1: Decision-making regarding the timing of invasive treatment initiation for patients with non-ST-segment elevation acute coronary syndrome (NSTE-ACS).

[0139] In this example, the sample patients consist of 25,173 adult patients from Hospital A in China and 2,946 adult patients from a publicly available US intensive care medical database. The baseline time is the admission time, and a decision point is determined every 6 hours. This example compares the predicted medical procedures output by the candidate model with the actual medical procedures prescribed by human doctors.

[0140] Figure 2This is a schematic diagram illustrating the comparison of multiple outcome effects of patient samples from Hospital A following a candidate model and medical operation data provided by human doctors in scenario 1, as provided in an embodiment of this disclosure. Figure 3 This is a schematic diagram comparing the probability of no-event outcomes in bleeding outcomes for a sample of patients in Hospital A following a candidate model and medical operation data provided by human doctors, as provided in an embodiment of this disclosure in Scenario 1. Figure 4 This is a schematic diagram showing the comparison of the probability of no events in major adverse cardiovascular events for a sample of patients in Hospital A following a candidate model and medical operation data provided by human doctors, as provided in an embodiment of this disclosure in Scenario 1.

[0141] like Figure 2 The evaluation results showed that, for the sample patients in Hospital A, following the recommended medical procedures provided by the candidate model resulted in a reduced risk of major adverse cardiovascular events during hospitalization compared to following real medical decisions made by human physicians, with an odds ratio of 0.74 (95% confidence interval: 0.62–0.87). Simultaneously, the odds ratio for grade 2–5 bleeding as defined by the Bleeding Academic Research Consortium was 0.78 (95% confidence interval: 0.62–0.94). Grade 2 is considered minor bleeding, and grades 3–5 are major bleeding. Figure 3 As shown, for sample patients in hospital A, following the recommended medical procedures provided by the candidate model reduced the probability of bleeding compared to following the actual medical decisions of human doctors. Figure 4 As shown, for sample patients in hospital A, following the recommended medical procedures provided by the candidate model reduced the probability of adverse cardiovascular events compared to following the actual medical decisions of human doctors.

[0142] Figure 5 This is a schematic diagram illustrating the comparison of multiple outcome effects of sample patients following candidate models and medical operation data provided by human doctors in the intensive care medical database under scenario 1, as provided in an embodiment of this disclosure. Figure 6 This is a schematic diagram comparing the probability of no-event outcomes in bleeding outcomes for sample patients in the intensive care medical database following the candidate model and the medical operation data provided by human doctors in Scenario 1, as provided in this embodiment of the disclosure. Figure 7 This is a schematic diagram showing the comparison of the probability of no major adverse cardiovascular events in the sample patients in the intensive care medical database following the candidate model and the medical operation data provided by human doctors in Scenario 1, as provided in the embodiments of this disclosure.

[0143] like Figure 5The evaluation results showed that, for sample patients in the intensive care medical database, following the recommended medical procedures provided by the candidate model resulted in a bleeding event odds ratio of 0.50 (95% confidence interval: 0.30–0.71) and a major adverse cardiovascular event odds ratio of 0.88 (95% confidence interval: 0.67–1.10) compared to following real medical decisions made by human physicians. Figure 6 As shown, for sample patients in the intensive care medical database, following the recommended medical procedures provided by the candidate model reduced the probability of bleeding compared to following the actual medical decisions of human doctors. Figure 7 As shown, for sample patients in the intensive care medical database, following the recommended medical procedures provided by the candidate model reduced the probability of major adverse cardiovascular events compared to following the actual medical decisions of human doctors.

[0144] Scenario 2: The decision on when to initiate prophylactic anticoagulation for hospitalized cancer patients.

[0145] Using the admission time of each of the 13,193 sample patients as the baseline, a decision point was determined every 24 hours to achieve daily dynamic assessment of the sample patients' adherence to the predicted medical procedures provided by the candidate model. The results showed that, compared with actual medical procedures (following human doctors), following the recommended medical procedures provided by the candidate model resulted in an odds ratio of 0.95 for venous thromboembolic events (95% confidence interval: 0.84–1.07) and an odds ratio of 0.91 for major bleeding (95% confidence interval: 0.80–1.01), demonstrating a trend towards causal benefit. Therefore, the method disclosed in this paper is applicable not only to acute cardiovascular scenarios but also to offline assessments of dynamic intervention scenarios such as prophylactic anticoagulation.

[0146] Scenario 3: The dynamic adjustment of anticoagulation and antithrombotic regimens for patients with atrial fibrillation complicated by acute coronary syndrome or after percutaneous coronary intervention.

[0147] For a sample of 2519 patients, the baseline time point was 24 hours after the diagnosis of atrial fibrillation complicated with acute coronary syndrome, or 24 hours after percutaneous coronary intervention. A decision time point was determined every 24 hours. The results showed that following the recommended medical procedures provided by the candidate model compared to actual medical procedures (following human doctors) resulted in a major adverse cardiovascular event odds ratio of 0.88 (95% confidence interval: 0.57–1.32) and an odds ratio of 0.98 (95% confidence interval: 0.80–1.17) for grade 2–5 bleeding as defined by the Bleeding Academic Research Consortium, also showing a positive trend. These results further demonstrate that the method disclosed in this paper can be used for estimating the value of candidate models in multi-level, dynamically switching clinical decision problems.

[0148] Figure 8This is a schematic diagram of the structure of a medical auxiliary decision-making evaluation system based on real-world longitudinal data, provided in an embodiment of this disclosure. The system includes: a time synchronization unit, a state encoding unit, a real operation mapping unit, a decision time point generation unit, a model output mapping unit, a structured matching unit, an evaluation unit, and a ranking output unit.

[0149] The time synchronization unit is used to align the data of each sub-item in the overall health data of the same sample patient from different sources to the same time axis, and to fill in the missing data sub-items and timestamps to obtain optimized overall health data.

[0150] The status coding unit is used to unify the names of identical sub-items in optimized overall health data from different sources, or to encode and map identical sub-items based on the same coding system. Furthermore, it extracts health status data and medical outcome data from the optimized overall health data.

[0151] The Real Operation Mapping Unit is used to extract real medical operation data from the optimized overall health data.

[0152] The decision-time point generation unit is used to generate decision-time points based on the disease progression, and aligns the timestamps of health status data, actual medical operation data, and medical outcome data with the decision-time points to obtain the health data corresponding to each decision-time point, i.e., the health data time series. The health data time series includes: health status time series, actual medical operation time series, and medical outcome time series. Furthermore, it can generate model-processing data based on the health data time series. The model-processing data includes historical health data corresponding to the decision-time points to be predicted.

[0153] The model output mapping unit is used to input the data to be processed into the candidate model. The candidate model outputs the predicted medical operation data corresponding to the decision point to be predicted. The predicted medical operation data may include: the timestamp of the occurrence of the medical operation.

[0154] The structured matching unit is used to determine, for sample patients, a data similarity index between actual and predicted medical operation data at the target decision-making point in time, and / or to encode the actual and predicted medical operation data at the target decision-making point in time into a structured vector. The structured matching unit can output the data similarity index at the target decision-making point in time, and / or the structured vectors of the predicted and actual medical operation data at the target decision-making point in time.

[0155] The evaluation unit is used to process data similarity indicators of multiple sample patients at multiple target decision points, and / or predicted medical operation data and actual medical operation data at each decision point, based on a time-varying causal inference algorithm, to obtain value estimation results.

[0156] The sorting output unit is used to sort the candidate models based on the value estimation results of each candidate model and output the sorting results.

[0157] Figure 9 A flowchart illustrating the process of determining data similarity indicators provided in this embodiment of the disclosure.

[0158] In this example, k comparison dimensions can be determined for the real medical operation data and the predicted medical operation data to be compared, where k is a positive integer. For each comparison dimension, a dimensional similarity index can be determined based on the following principles: class constraint matching, terminology (encoding mapping) matching, or intensity level matching. The class constraint matching principle explains the principle for determining the similarity between two things (medical operations) belonging to the same class. The terminology (encoding mapping) matching principle indicates that events belonging to the same category or with the same code are completely identical. The intensity level matching principle explains the mapping relationship between the intensity level of two similar medical operations and the similarity between the two operations.

[0159] The comparison dimension can be determined from the fields of the medical operation data based on the type of medical operation. Fields can be used directly as comparison dimensions. Alternatively, multiple related fields can be combined to form a field combination, which can then be used as the comparison dimension.

[0160] In this example, the first comparison dimension consisting of a single field can be determined, the first dimension similarity index of the first comparison dimension can be determined, and the first dimension similarity indexes can be weighted and aggregated to obtain the data similarity index between real medical operation data and predicted medical operation data.

[0161] Alternatively, a second comparison dimension composed of multiple fields can be determined, a second-dimensional similarity index for the second comparison dimension can be determined, and the second-dimensional similarity indexes can be weighted and aggregated to obtain the data similarity index between real medical operation data and predicted medical operation data.

[0162] Alternatively, if both a first comparison dimension and a second comparison dimension exist, a weighted aggregation can be performed on the similarity indicators of each first dimension and each similarity indicator of each second dimension.

[0163] Data similarity indicators can be represented as continuous numerical values, discrete level labels, or multidimensional vectors.

[0164] Figure 10 This is a schematic flowchart illustrating the evaluation method for candidate models provided in an embodiment of this disclosure.

[0165] In this example, an evaluation model is used to assess the medical decision-making support capability of a candidate model. This evaluation model can use an inverse probability weighted algorithm to process the data similarity indicators, covariate information, and medical outcome data of multiple sample patients at the target decision time point, obtaining a first value estimate of the candidate model. Alternatively, the model can use the G-Formula algorithm to process the predicted medical procedure data, actual medical procedure data, covariate information, and medical outcome data of multiple sample patients at the target decision time point, obtaining a second value estimate. Alternatively, the evaluation model can use a combination of the inverse probability weighted algorithm and the G-Formula algorithm to process the predicted medical procedure data, actual medical procedure data, covariate information, and medical outcome data of each sample patient at the target decision time point, obtaining a third value estimate. The covariate information includes one or more of the following: patient medical history, demographic characteristics, baseline health status, comorbidity information, previous treatment history, and time-series health status characteristics.

[0166] Figure 11 A schematic diagram of the structure of a medical auxiliary decision-making and evaluation device based on real-world longitudinal data provided in an embodiment of this disclosure. Figure 11 As shown, the device 20 includes:

[0167] The health data time series determination unit 21 is used to determine the health data of the sample patient at each decision time point, and obtain the health data time series of the sample patient. The health data includes: health status data, actual medical operation data, and medical outcome data.

[0168] The historical health data determination unit 22 is used to determine the historical health data corresponding to the decision point to be predicted based on the health data time series.

[0169] The model data to be processed determination unit 23 is used to determine the model data to be processed, which includes historical health data corresponding to the decision point to be predicted.

[0170] The predictive medical operation data determination unit 24 is used to input the model to be processed data into the candidate model to obtain the predictive medical operation data corresponding to the decision point to be predicted.

[0171] The value estimation result determination unit 25 is used to process the predicted medical operation data, real medical operation data, covariate information, and medical outcome data of multiple sample patients at multiple target decision time points based on a time-varying causal inference algorithm, so as to obtain the value estimation result for evaluating the value of the auxiliary medical decision-making strategy of the candidate model. The covariate information represents the baseline and time-varying conditions of the sample patients.

[0172] In one possible implementation, the value estimation result determination unit 25 is further configured to:

[0173] For each sample of patients, a data similarity index is determined between the actual medical operation data and the predicted medical operation data corresponding to the target decision point;

[0174] Based on the inverse probability weighted algorithm, the data similarity indicators, covariate information, and medical outcome data of multiple sample patients at the multiple target decision time points are processed to obtain the first value estimation result;

[0175] or,

[0176] Based on the G-Formula algorithm, the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of multiple sample patients at multiple target decision points are processed to obtain a second value estimation result.

[0177] or,

[0178] Based on a combination of the inverse probability weighted algorithm and the G-Formula algorithm, the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of each sample patient at the multiple target decision points are processed to obtain a third value estimation result.

[0179] In one possible implementation, the value estimation result determination unit 25 is further configured to:

[0180] Based on the type of medical operation, a comparison dimension is determined between the actual medical operation data and the predicted medical operation data at the target decision point, and the comparison dimension is determined by the fields in the medical operation data.

[0181] Determine the dimensional similarity indicators between real medical operation data and predicted medical operation data in each comparison dimension;

[0182] The data similarity indicators are determined based on the similarity indicators of each dimension.

[0183] In one possible implementation, the health data time series determination unit 21 is further configured to:

[0184] Based on the disease course of the sample patients, multiple decision-making time points were determined;

[0185] Obtain the overall health data of the sample patients, wherein each sub-item of the overall health data includes a timestamp;

[0186] Identify abnormal data in overall health data;

[0187] Based on the anomaly type of the abnormal data, different strategies are used to correct the abnormal data and obtain optimized overall health data.

[0188] To optimize overall health data, health data corresponding to each decision point is determined based on timestamps and decision point alignment.

[0189] In one possible implementation, the exception types include: multiple timestamps for the same event, timestamp conflicts, missing timestamps, and missing data;

[0190] The health data time series determination unit 21 is further used for:

[0191] In the case where the abnormal data consists of multiple timestamps of the same event, the target timestamp is determined from multiple candidate timestamps based on the event to which the abnormal data belongs, according to a preset rule.

[0192] In the case of timestamp conflicts in the abnormal data, the conflicting timestamps are corrected based on the adjacent events and / or the hospitalization stage to which the abnormal data belongs;

[0193] In the case where the abnormal data is missing timestamps or missing data, the missing timestamps or missing data are supplemented based on the events adjacent to the abnormal data and / or the hospitalization stage to which they belong.

[0194] In one possible implementation, the model data determination unit 23 is further used for:

[0195] Historical health data corresponding to the decision-making time point to be predicted is used as the data to be processed by the model.

[0196] or,

[0197] The actual medical operation data and historical health data corresponding to the decision-making time point to be predicted are used as the data to be processed by the model.

[0198] In one possible implementation, the predictive medical operation data determination unit 24 is further configured to:

[0199] Input the historical health data corresponding to the decision point to be predicted into the candidate model;

[0200] The prompt words are input into the candidate model so that the candidate model can obtain the predicted medical operation data corresponding to the decision point to be predicted based on the historical health data of the decision point to be predicted.

[0201] In one possible implementation, the health status data at least represents: physical indicator values, medical event records, source system, reference interval, anomaly markers, evidence locations, missing markers, and timestamps corresponding to the physical indicator values ​​and the medical event records;

[0202] The actual medical operation data includes one or more of the following: medical operation name, standard terminology code of medical operation, medical operation timestamp, medical operation type, target or site of action, dosage or intensity, frequency, route, and course of treatment;

[0203] Medical outcome data should include at least one of the following: disease progression, complications, in-hospital adverse events, discharge mode, readmission, death, functional recovery, and medical costs.

[0204] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0205] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0206] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0207] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0208] Figure 12 This is a schematic diagram of an electronic device for medical decision-making assessment based on real-world longitudinal data, provided as an embodiment of this disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. (Refer to...) Figure 12 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0209] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TMMac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0210] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0211] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0212] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0213] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0214] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0215] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0216] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0217] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0218] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A medical decision support evaluation method based on real-world longitudinal data, characterized in that, include: Determine the health data of the sample patients at each decision point to obtain the time series of the health data of the sample patients. The health data includes: health status data, actual medical operation data, and medical outcome data. Based on the health data time series, determine the historical health data corresponding to the decision point to be predicted; The model needs to process data, which includes historical health data corresponding to the decision point to be predicted. Input the data to be processed in the model into the candidate model to obtain the predicted medical operation data corresponding to the decision point to be predicted. Based on a time-varying causal inference algorithm, the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of multiple sample patients at multiple target decision points are processed to obtain a value estimation result for evaluating the value of the auxiliary medical decision-making strategy of the candidate model. The covariate information represents the baseline and time-varying conditions of the sample patients.

2. The method according to claim 1, characterized in that, Based on a time-varying causal inference algorithm, the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of multiple sample patients at multiple target decision points are processed to obtain a value estimation result for evaluating the auxiliary medical decision-making strategy value of the candidate model, including: For each sample of patients, a data similarity index is determined between the actual medical operation data and the predicted medical operation data corresponding to the target decision point; Based on the inverse probability weighted algorithm, the data similarity indicators, covariate information, and medical outcome data of multiple sample patients at the multiple target decision time points are processed to obtain the first value estimation result; or, Based on the G-Formula algorithm, the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of multiple sample patients at multiple target decision points are processed to obtain a second value estimation result. or, Based on a combination of the inverse probability weighted algorithm and the G-Formula algorithm, the predicted medical operation data, actual medical operation data, covariate information, and medical outcome data of each sample patient at the multiple target decision points are processed to obtain a third value estimation result.

3. The method according to claim 2, characterized in that, For each sample of patients, data similarity indicators are determined between the actual medical operation data and the predicted medical operation data corresponding to the target decision-making time point, including: Based on the type of medical operation, a comparison dimension is determined between the actual medical operation data and the predicted medical operation data at the target decision point, and the comparison dimension is determined by the fields in the medical operation data. Determine the dimensional similarity indicators between real medical operation data and predicted medical operation data in each comparison dimension; The data similarity indicators are determined based on the similarity indicators of each dimension.

4. The method according to claim 1, characterized in that, The determined health data of the sample patients at each decision point includes: Based on the disease course of the sample patients, multiple decision-making time points were determined; Obtain the overall health data of the sample patients, wherein each sub-item of the overall health data includes a timestamp; Identify abnormal data in overall health data; Based on the anomaly type of the abnormal data, different strategies are used to correct the abnormal data and obtain optimized overall health data. To optimize overall health data, health data corresponding to each decision point is determined based on timestamps and decision point alignment.

5. The method according to claim 4, characterized in that, The anomaly types include: multiple timestamps for the same event, timestamp conflicts, missing timestamps, and missing data. The abnormal data, based on different anomaly types, uses different strategies to correct the abnormal data, resulting in optimized overall health data, including: In the case where the abnormal data consists of multiple timestamps of the same event, the target timestamp is determined from multiple candidate timestamps based on the event to which the abnormal data belongs, according to a preset rule. In the case of timestamp conflicts in the abnormal data, the conflicting timestamps are corrected based on the adjacent events and / or the hospitalization stage to which the abnormal data belongs; In the case where the abnormal data is missing timestamps or missing data, the missing timestamps or missing data are supplemented based on the events adjacent to the abnormal data and / or the hospitalization stage to which they belong.

6. The method according to claim 1, characterized in that, The determined model data to be processed includes: Historical health data corresponding to the decision-making time point to be predicted is used as the data to be processed by the model. or, The actual medical operation data and historical health data corresponding to the decision-making time point to be predicted are used as the data to be processed by the model.

7. The method according to claim 1, characterized in that, The step of inputting the data to be processed into the candidate model to obtain the predicted medical operation data corresponding to the decision point to be predicted includes: Input the historical health data corresponding to the decision point to be predicted into the candidate model; The prompt words are input into the candidate model so that the candidate model can obtain the predicted medical operation data corresponding to the decision point to be predicted based on the historical health data of the decision point to be predicted.

8. The method according to any one of claims 1-7, characterized in that, The health status data at least represents: physical indicator values, medical event records, source system, reference range, anomaly markers, evidence locations, missing markers, and timestamps corresponding to the physical indicator values ​​and the medical event records; The actual medical operation data includes one or more of the following: medical operation name, standard terminology code of medical operation, medical operation timestamp, medical operation type, target or site of action, dosage or intensity, frequency, route, and course of treatment; Medical outcome data should include at least one of the following: disease progression, complications, in-hospital adverse events, discharge mode, readmission, death, functional recovery, and medical costs.

9. A medical auxiliary decision-making and evaluation device based on real-world longitudinal data, characterized in that, The device includes: A health data time series determination unit is used to determine the health data of the sample patients at each decision point, thereby obtaining the health data time series of the sample patients. The health data includes: health status data, actual medical operation data, and medical outcome data. The historical health data determination unit is used to determine the historical health data corresponding to the decision point to be predicted based on the health data time series. The model data to be processed determination unit is used to determine the model data to be processed, which includes historical health data corresponding to the decision point to be predicted. The predictive medical operation data determination unit is used to input the data to be processed in the model into the candidate model to obtain the predictive medical operation data corresponding to the decision point to be predicted. The value estimation result determination unit is used to process the predicted medical operation data, real medical operation data, covariate information, and medical outcome data of multiple sample patients at multiple target decision time points based on a time-varying causal inference algorithm, so as to obtain the value estimation result for evaluating the value of the auxiliary medical decision-making strategy of the candidate model. The covariate information represents the baseline and time-varying conditions of the sample patients.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

Images