Methods, programs, and systems for generating outcome definitions

Abstract

This invention provides a method for generating an outcome definition that has a certain degree of validity for a given outcome. [Solution] The method includes: step 110 preparing a number of training data sets, each containing data that encodes one or more medical records of a patient and data that encodes whether a certain outcome is occurring in the patient; and step 120 generating a model that infers whether a certain outcome is occurring in a patient by machine learning using the multiple training data sets, and identifying one or more medical records that contribute to the classification performance of whether a certain outcome is occurring in a patient. The model takes at least data that encodes a patient's medical records as input and outputs at least data that encodes whether a certain outcome is occurring in a patient. The method also includes step 140 generating outcome definitions based on the identified one or more medical records.

Description

Technical Field

[0001] The present disclosure relates to generating outcome definitions for a given outcome.

Background Art

[0002] An "outcome" refers to an evaluation item used to assess the effect of medical interventions such as treatment or prevention, for example in clinical research, and includes outcomes such as death, morbidity, and results during the course of testing. For example, when non-steroidal anti-inflammatory drugs (NSAIDs) such as aspirin or loxoprofen are prescribed to suppress inflammation, pain, and fever, if it is to be determined whether the proportion of patients who develop kidney damage as a side effect (adverse event) of the therapeutic drug has increased, kidney damage is the "outcome" in this case.

[0003] Furthermore, an "outcome definition" refers to the conditions that define an "outcome" or the patient's condition or changes. When death is taken as the "outcome", examples of the corresponding "outcome definition" include "when a physician determines that death has occurred" or "when all of cardiac pulse arrest, respiratory arrest, pupil dilation, and loss of the pupillary light reflex are present", and various "outcome definitions" can be set for a single "outcome".

Prior Art Literature

Non-Patent Literature

[0004]

Non-Patent Literature 1

Summary of the Invention

[0005] For example, if we use cerebral hemorrhage as an outcome and try to identify patients who have suffered it in a database, defining the outcome as "cerebral hemorrhage recorded in the hospitalization diagnosis and with records of tests such as CT scans" will capture most patients who actually had a cerebral hemorrhage, but will also capture many patients who did not actually have one. This is because many people who were diagnosed with a cerebral hemorrhage underwent tests for it, but were not found to have one. In healthcare, a diagnosis of cerebral hemorrhage is required before undergoing tests for it, resulting in this type of record. However, by adding conditions such as "prescription of antihypertensive drugs" to this definition, the results will be narrowed down to patients who were actually diagnosed with a cerebral hemorrhage, thus reducing false positives. In this way, when evaluating outcomes from medical records in a medical information database, it is necessary to devise an outcome definition that allows for highly accurate evaluation. [Means for solving the problem]

[0006] This disclosure has been made in view of the above, and one of its objectives is to provide a method, etc., capable of generating an outcome definition with a certain degree of validity for a given outcome.

[0007] Embodiments of the present disclosure provide a method for generating an outcome definition for a certain outcome, comprising the steps of: preparing a plurality of training data, each training data including data encoding one or more medical records of a patient and data encoding whether the certain outcome is occurring in the patient; generating a model for inferring whether the certain outcome is occurring in a patient by machine learning using the plurality of training data, and identifying one or more medical records that contribute to the classification performance of whether the certain outcome is occurring in the patient, wherein the model takes at least data encoding the medical records of the patient as input and at least data encoding whether the certain outcome is occurring in the patient as output; and generating the outcome definition based on the identified one or more medical records.

[0008] In one embodiment, the one or more medical records containing encoded data in each training data set may not include a certain clinical test value, and the outcome definition generated therefrom may not use the aforementioned clinical test value.

[0009] In one embodiment, the generated outcome definition may include at least one of the following: a diagnosis was made, a drug was administered, and a medical procedure was performed.

[0010] In one embodiment, the generated outcome definition may further include that a clinical test value other than the given clinical test value is greater than, less than, greater than or equal to, or less than or equal to a predetermined value.

[0011] In one embodiment, the step of preparing a plurality of training data may include the steps of: obtaining one or more medical records of a patient from a first database, which include one or more medical records of the patient that include a certain clinical test value; generating data that encodes one or more medical records of the patient that do not include a certain clinical test value, based on the obtained one or more medical records; and generating data that encodes whether the certain outcome is expressed in the patient, based on the obtained one or more medical records, according to a known outcome definition for the certain outcome that uses the certain clinical test value.

[0012] In one embodiment, the machine learning includes a gradient boosting method, the resulting model may include a plurality of decision trees, and the one or more medical records identified as contributing to the classification performance of whether a certain outcome occurs in a particular patient may include at least one of the medical records used by the plurality of decision trees as branching conditions.

[0013] In one embodiment, the step of preparing a plurality of training data may include the steps of: calculating a Shap value for each medical record using a predetermined model that infers whether a certain outcome occurs in a certain patient, which is different from the generated model, wherein the predetermined model takes at least data encoded from the medical record of the certain patient as input and at least data encoded from whether the certain outcome occurs in the certain patient as output; and preparing the plurality of training data such that the predetermined training data is not included, wherein the predetermined training data is training data that includes data encoded from a predetermined medical record, and the predetermined medical record is a medical record in which the Shap value is less than or equal to a predetermined value, or is not included in a predetermined number of medical records with the largest Shap values.

[0014] In one embodiment, the step of generating a model that infers whether a certain outcome is occurring in a patient by machine learning using the plurality of training data, and identifying one or more medical records that contribute to the classification performance of whether a certain outcome is occurring in a patient, may include the steps of using the generated model to calculate an external evaluation criterion for each medical record, and identifying the one or more medical records based on the external evaluation criterion.

[0015] In one embodiment, the external evaluation criterion may be a Shap value, and the step of identifying one or more medical records based on the external evaluation criterion may include the step of selecting one or more medical records whose Shap value is greater than or equal to a predetermined value, or selecting a predetermined number of medical records from the largest Shap values ​​to be identified.

[0016] According to embodiments of the present disclosure, a method is provided for using an outcome definition for a certain outcome that does not use a certain clinical test value, the method comprising the steps of: generating the outcome definition that does not use a certain clinical test value by the method described above; obtaining one or more medical records of a patient from a second database; determining whether the certain outcome is occurring in the patient based on the obtained one or more medical records in accordance with the generated outcome definition; and adding information indicating whether the certain outcome is occurring in the patient to the second database.

[0017] In one embodiment, the aforementioned outcome is the development of diabetic nephropathy, the aforementioned clinical test value is eGFR, and the information indicating whether the aforementioned outcome is present in the patient may, if the aforementioned outcome is present in the patient, be information indicating that the patient has developed diabetic nephropathy or that the eGFR is less than 30, and if the aforementioned outcome is not present in the patient, be information indicating that the patient does not have diabetic nephropathy or that the eGFR is 30 or higher.

[0018] In one embodiment, the medical record may include a medical treatment record, a clinical record, a diagnostic record, a drug administration record, a medical treatment record, and a patient background.

[0019] According to embodiments of the present disclosure, a program is provided for generating an outcome definition for a certain outcome, the program causing a computer to perform the following steps: preparing a plurality of training data, each training data including data encoding one or more medical records of a patient and data encoding whether the certain outcome is occurring in the patient; generating a model that infers whether the certain outcome is occurring in a patient by machine learning using the plurality of training data, and identifying one or more medical records that contribute to the classification performance of whether the certain outcome is occurring in the patient, wherein the model takes at least data encoding the medical records of the patient as input and at least data encoding whether the certain outcome is occurring in the patient as output; and generating the outcome definition based on the identified one or more medical records.

[0020] According to embodiments of the present disclosure, a system is provided for generating an outcome definition for a certain outcome, comprising the steps of: preparing a plurality of training data, each training data including data encoding one or more medical records of a patient and data encoding whether the certain outcome is occurring in the patient; generating a model that infers whether the certain outcome is occurring in a patient by machine learning using the plurality of training data, and identifying one or more medical records that contribute to the classification performance of whether the certain outcome is occurring in the patient, wherein the model takes at least data encoding the medical records of the patient as input and at least data encoding whether the certain outcome is occurring in the patient as output; and generating the outcome definition based on the identified one or more medical records. Effects of the Invention

[0021] According to an embodiment of the present disclosure, an outcome definition having a certain degree of validity for a certain outcome can be generated. Brief Description of the Drawings

[0022] [Figure 1A] A flowchart of an exemplary method for generating an outcome definition for a certain outcome. [Figure 1B] A diagram illustrating an exemplary decision tree. [Figure 1C] A flowchart of exemplary processing that can be included in step 120. [Figure 2A] A flowchart of exemplary processing for preparing a plurality of teacher data. [Figure 2B] A flowchart of exemplary processing for preparing a plurality of teacher data. [Figure 3] An exemplary graph in which the vertical axis corresponds to medical records and the horizontal axis corresponds to their Shap values. [Figure 4] A flowchart of an exemplary method for using an outcome definition for a certain outcome. [Figure 5] A configuration diagram of an exemplary system for generating an outcome definition for a certain outcome. [Figure 6] A diagram illustrating an example of the hardware configuration of a computer. Mode for Carrying Out the Invention

[0023] 1 Method for generating outcome definition Figure 1A is a flowchart of an exemplary method 100 for generating an outcome definition for a certain outcome, according to one embodiment of the present disclosure. The exemplary method 100 may be executed by a computer, and therefore each step included in the exemplary method 100 may also be executed by a computer. Furthermore, an example of "a certain outcome" is "having been diagnosed with diabetic nephropathy," but is not limited to this.

[0024] Example method 100 utilizes machine learning with training data, and 110 shows the steps for preparing multiple training data for machine learning. Each training data includes data encoded from one or more medical records of a patient and data encoded on whether the patient exhibits the aforementioned outcome.

[0025] Medical records may be, but are not limited to, medical records or clinical records. Furthermore, a single medical record may be, but is not limited to, a record of diagnosis, a record of drug administration, or a record of medical treatment. In addition, a single medical record may also be a record of patient background. Specifically, a single medical record may, but is not limited to, the following: • Diagnosed with chronic renal failure • Diagnosed with diabetes • Febuxostat (a medication for treating hyperuricemia) was administered. • Dialysis was performed.

[0026] An example of data encoding one or more medical records of a patient is the feature quantities of each of multiple, preferably a vast number of, separate medical records. More specifically, an example of data encoding one or more medical records of a patient is a vector in which each element is a feature quantity of multiple, preferably a vast number of, separate medical records. In such a vector, for example, the element whose feature quantity is "Feburic was administered" may take the value 1 if the one or more medical records in question contain the medical record "Feburic was administered," and 0 if they do not. Even if the one or more medical records in question contain the medical record "Feburic was administered" multiple times, the element whose feature quantity is "Feburic was administered" may take the value 1. Furthermore, such a vector is merely an example, and data encoding one or more medical records is not limited to this.

[0027] Examples of data encoding whether the above-mentioned outcome is expressed in the patient in each training data set include, but are not limited to, values ​​that take 1 if the outcome is expressed and 0 if it is not, or vectors that include such values ​​as elements.

[0028] Step 110 may include, but is not limited to, the step of reading a given set of training data from the computer's auxiliary storage device or an external storage device into the computer's main memory, or it may include the example method 200 described later.

[0029] Paragraph 120 describes a step of generating a model that infers whether a certain outcome occurs in a particular patient using machine learning with multiple training datasets, and identifying one or more medical records that contribute to the performance of the generated model, i.e., the classification performance of whether a certain outcome occurs in a particular patient. This model takes at least data encoded from the patient's medical records as input and outputs at least data encoded from whether a certain outcome occurs in that patient. From another perspective, machine learning using multiple training datasets may be machine learning in which data encoded from one or more medical records of a patient are independent variables and data encoded from whether a certain outcome occurs in that patient is the dependent variable.

[0030] An example of data encoded by the generated model indicating whether a certain outcome is expressed in a particular patient is a value that takes the value 1 if the outcome is expressed and 0 if it is not, or a vector containing such a value as an element. Another example of data encoded by the generated model indicating whether a certain outcome is expressed in a particular patient is a value indicating the likelihood of expression (for example, a value closer to 1 if the likelihood is high, and closer to 0 if the likelihood is low), or a vector containing such a value as an element. The data encoded by the generated model indicating whether a certain outcome is expressed in a particular patient is not limited to these examples.

[0031] The machine learning example in Step 120 is XGBoost, but it is not the only machine learning method used. Other machine learning methods that may be used in Step 120 are listed below. • Linear regression (multivariate regression analysis, analysis of variance, analysis of covariance, partial least squares regression, ridge regression analysis, Lasso regression analysis, logistic regression analysis, etc.) • Discriminant analysis (linear discriminant analysis, canonical discriminant analysis, partial least squares discriminant analysis, etc.) • Ensemble learning (e.g., Random Forest, LightGBM; XGBoost belongs to this category.) • Neural networks (convolutional neural networks, recurrent neural networks, variational autoencoders, generative adversarial networks, etc.) • Support vector machines, canonical correlation analysis, etc.

[0032] The method for identifying one or more medical records that contribute to the classification performance of the generated model, or in other words, the method for determining whether a particular medical record contributes to that classification performance, is arbitrary, but one example will be described later.

[0033] Paragraph 140 illustrates the step of generating an outcome definition for a given outcome based on one or more identified medical records. The generated outcome definition may include at least one of the following: a diagnosis was made (condition), a drug was administered (condition), or a medical procedure was performed (condition).

[0034] More specifically, an example of a generated outcome definition is a combination of one or more conditions, where each of the one or more conditions may correspond to each of the one or more identified medical records. Diagnosed with kidney failure, and, A medication indicated for renal failure has been prescribed, and Medical procedures performed on patients with renal failure are carried out, and …(The ellipsis indicates an omission.)

[0035] Another example of a generated outcome definition is a classifier or a program that causes a computer to function as such a classifier to determine whether a certain outcome is occurring in a patient according to one or more combinations of conditions as described above. This classifier may take data encoded from one or more medical records of a patient as input and output data encoded to indicate whether the patient is experiencing the outcome described above. An example of the data encoded from one or more medical records of a patient to be input to this classifier is a vector containing only elements whose features are each of the identified one or more medical records, but is not limited to this. Similarly, an example of the data encoded to indicate whether the patient is experiencing the outcome described above, output by this classifier, is similar to, but is not limited to, the output of the model generated in step 120.

[0036] This classifier may be constructed by modifying the generated model described above. For example, the classifier described above can be constructed by adding a module to the front of the generated model that generates a vector to be input to the model (hereinafter referred to as the "second vector") from a vector (hereinafter referred to as the "first vector") that contains only elements whose features are each of the one or more identified medical records described above. This module may operate by copying the values ​​of each element of the first vector to the corresponding elements of the second vector, and then setting the elements of the second vector that do not have corresponding elements in the first vector to a predetermined value (for example, 0).

[0037] Note that the outcome definition generated in step 140 is not limited to this.

[0038] 2. Preparation of multiple training data sets Figure 2A is a flowchart of an example process 200 for preparing multiple training data sets, which may be included in step 110.

[0039] Paragraph 210 shows the step of obtaining one or more medical records from the first database, which include a certain clinical test value for one or more patients. Therefore, the first database is a database that includes the above-mentioned clinical test value as a medical record for one or more patients.

[0040] Step 220 indicates a step of generating data by encoding one or more medical records of the patient that do not include the above-mentioned clinical test value, based on the one or more medical records obtained. Step 220 may include a step of generating one or more new medical records by excluding the above-mentioned clinical test value from the one or more medical records obtained, and then encoding this new one or more medical records to generate data.

[0041] 230 is a known outcome definition for a certain outcome, and it describes a step of generating data that encodes whether the certain outcome is occurring in the patient, based on one or more medical records obtained, according to a known outcome definition using a certain clinical test value.

[0042] One example of a clinical test value is eGFR, but it is not limited to this. Similarly, an example of a known outcome definition using the above-mentioned clinical test value is an eGFR of less than 30 (condition), but it is not limited to this. Note that an eGFR of less than 30 is a publicly recognized and validated outcome definition for diabetic nephropathy.

[0043] The technical significance of example process 200 will be explained below.

[0044] Outcome definitions can be generated using various methods, but it is necessary to verify whether the generated outcome definitions can capture the actual disease state at a reasonable level for use in clinical research. Validation studies are typically conducted for this purpose.

[0045] In studies using medical information databases, it is recommended to create several to a dozen outcome definitions for the target outcome, rather than just one, in order to appropriately select the appropriate outcome definition from multiple candidates, and to calculate the positive predictive value and sensitivity for each outcome definition. In order to calculate the positive predictive value and sensitivity, it is necessary to identify patients (true cases) within the patient population being studied who actually meet the target outcome.

[0046] In determining a true case, a judgment form or similar document, which specifically and objectively describes the criteria for determining a case, is created in advance based on the opinions of specialists. With the cooperation of multiple medical facilities and many physicians, the true case is generally determined by carefully examining the contents of the medical record. This examination of the medical record does not simply involve checking for the presence or absence of a disease name; it also involves re-examining records of symptoms, images, and other clinical findings. The determination of whether a case is true is made independently of the diagnosis at the time the medical record was recorded. Furthermore, it is considered desirable, in principle, for each case to be independently judged by two or more specialists. If independent judgment by two or more specialists is difficult, true cases can be identified through methods such as a non-specialist physician making the initial judgment followed by a specialist's final judgment, or a non-physician healthcare professional collecting the necessary information for the judgment while a specialist makes the final decision. Thus, validation research to identify true cases based on medical records requires enormous costs and time.

[0047] On the other hand, in cases where the occurrence of an outcome can be objectively determined using only information recorded in a medical information database, such as outcomes that can be defined solely by clinical test values, that information can be used directly, and validation is not required.

[0048] In other words, according to example processing 200, it is possible to determine whether a true case, i.e., a patient, is experiencing an outcome using a publicly recognized and valid outcome definition that can be defined solely by clinical test values. This avoids the considerable effort that can be required when preparing multiple training data sets, as described above.

[0049] Furthermore, according to the example process 200, the validity of whether the above outcome occurs in patients whose training data includes encoded data in each training data set is ensured. Therefore, outcome definitions that differ from known outcome definitions, generated in step 140, can also be validated in the same way without conducting validation studies.

[0050] Furthermore, if step 110 of the exemplary method 100 includes exemplary processing 200, it will be understood that one or more medical records of a patient in which the encoded data is included in each of the multiple training data prepared in step 110 will not include the aforementioned clinical test value. However, even if step 110 does not include exemplary processing 200 and multiple training data are given, it will be understood that one or more medical records of a patient in which the encoded data is included in each of such multiple training data may not include the aforementioned clinical test value.

[0051] Furthermore, it will be understood that the outcome definition generated in step 140 of the example method 100 will not use the aforementioned clinical test value, because one or more medical records of patients containing the encoded data in each of the multiple training data used to generate the outcome definition will not include the aforementioned clinical test value.

[0052] Furthermore, the fact that one or more of a patient's medical records do not contain the aforementioned clinical test values ​​does not mean that one or more of a patient's medical records do not contain any clinical test values ​​at all. That is, one or more of a patient's medical records may not contain the aforementioned clinical test values, but may contain other clinical test values. Accordingly, the outcome definition generated in step 140 may include conditions such as the clinical test value other than the aforementioned clinical test value being greater than, less than, greater than or equal to, or less than or equal to a predetermined value.

[0053] Figure 2B is a flowchart of another exemplary process 250 for preparing multiple training data sets, which step 110 may include.

[0054] Step 255 shows a step of calculating a Shape value for each medical record using a predetermined model that infers whether a certain outcome is occurring in a particular patient. This predetermined model takes at least data encoded from a patient's medical record as input and at least data encoded from whether the patient is occurring as output. This predetermined model can be any model, but it is preferable that it is different from the model generated in step 120. Furthermore, when preparing multiple training data using the example process 250, it is preferable in step 120 to use a method that identifies one or more medical records that contribute to the classification performance of whether a certain outcome is occurring in a particular patient, without using Shape values. For more information on Shape values, please refer to Section 3.

[0055] Paragraph 260 shows a step of generating a second set of training data by excluding a predetermined set of training data from a first set of training data. This predetermined set of training data includes data encoded from a predetermined medical record. This predetermined medical record may be a medical record whose Shape value is less than or equal to a predetermined value. Alternatively, this predetermined medical record may be excluded from a predetermined number of medical records with the largest Shape values. Furthermore, the first set of training data may be given or may be prepared by example process 200.

[0056] 265 shows the step of making the generated second set of training data into the prepared set of training data.

[0057] Steps 260 and 265 will show that multiple training data sets are prepared so that none of the predetermined training data are included.

[0058] The technical significance of example process 250 will be explained below.

[0059] By pre-deleting training data containing encoded medical records (variables; features) that contribute little to classification performance based on a criterion called the Shap value, local optimization is avoided in the subsequent machine learning in step 120. This is expected to improve the classification performance of the final outcome definition (including the classifier as described above).

[0060] In particular, if, in the subsequent step 120, one or more medical records that contribute to the classification performance of whether a certain outcome is occurring in a particular patient are identified using criteria other than the Shap value, then, as a result, medical records that contribute to the classification performance are identified from multiple different criteria, which is expected to improve the classification performance of the final outcome definition.

[0061] Note that the Shap value is just one example of an external evaluation criterion calculated for medical records. Therefore, in example process 250, a different external evaluation criterion may be used instead of the Shap value. Examples of other external evaluation criteria include the Gain value, Cover value, and Weight (=Frequency) value. Note that an external evaluation criterion is a criterion for evaluating the importance of each medical record (variable; feature) in the model, and the external evaluation criteria are not limited to the Shap value, Gain value, Cover value, and Weight value mentioned above.

[0062] 3. Identification of medical records that contribute to classification performance. One or more medical records that contribute to the classification performance of whether a particular outcome occurs in a patient can be identified based on the parameters included in the generated model. For example, if a model including a decision tree is generated by machine learning, at least one of the medical records that the decision tree in the generated model uses as a branching condition can be identified as one or more medical records.

[0063] Figure 1B is a diagram representing an example decision tree 140. Each of the 141s represents one data point (which is training data in machine learning). 142 indicates a branch, and 143 indicates a branching condition, where "Medical Record A" and "Medical Record B" are medical records used as branching conditions in the example decision tree 140. 144 is the output (classification result) of the example decision tree 140, and the output of a model may be generated based on the outputs of multiple such decision trees included in the model. Note that this example decision tree 140 is simplified for illustrative purposes.

[0064] For example, in the gradient boosting method used in XGBoost, multiple decision trees are created during the model generation process. In summary, the medical records (variables; features) used as branching conditions in one decision tree are only a portion of all the encoded medical records (variables; features). Furthermore, the medical records (variables; features) used as branching conditions differ for each decision tree. Then, in gradient boosting, in order to improve classification performance, a new decision tree is created for the training data that failed to be classified in the previously created decision trees. The decision tree that satisfies the condition of minimizing the log loss value, which is the evaluation criterion for classification performance, is adopted, and some of the created decision trees are discarded. An example of the log loss value LogLoss, which is the evaluation criterion for classification performance, is shown below.

number

[0065] Through the repetition of the above process, the medical records used as branching conditions by the multiple decision trees included in the model finally generated by the gradient boosting method may be one or more medical records that contribute to the classification performance of whether a certain outcome is occurring in a particular patient.

[0066] Alternatively, one or more medical records that contribute to the classification performance of whether a particular outcome is occurring in a patient can be identified based on the Shap (SHapley Additive exPlanations) value of each medical record (variable; feature).

[0067] Figure 1C is a flowchart of an example process 150 that step 120 may include.

[0068] Reference 155 illustrates the step of generating a model that infers whether a particular outcome occurs in a patient using machine learning with multiple training datasets.

[0069] Paragraph 160 shows the step of calculating the Shap value for each medical record using the generated model described above. The Shap value is an application of the Shapley value from game theory to machine learning, and can be calculated using the generated model with known software or libraries.

[0070] Figure 3 is an example graph 300 in which the vertical axis represents medical records and the horizontal axis represents their Shape values. More specifically, if the data encoded from one or more patient medical records used in machine learning is a vector in which each element is a feature of multiple separate medical records, then each medical record appearing on the vertical axis may correspond to each element of that vector. However, in the example graph 300, the medical records are not arranged in the order of their corresponding elements in the vector, but rather in descending order of their Shape values. Note that the example graph 300 has been simplified for illustrative purposes.

[0071] In step 120, one or more medical records with a Shape value greater than or equal to a predetermined value may be considered to contribute to the model's classification performance. Step 165 (which may be included in step 120) shows the step of identifying one or more medical records with a Shape value greater than or equal to a predetermined value. In the example graph 300, 310 shows the predetermined value, and 320 shows the one or more medical records identified.

[0072] Alternatively, in step 120, a predetermined number of medical records with the largest Shape values ​​may be considered to contribute to the model's classification performance. Therefore, step 165 (which may include step 120) may be a step in which a predetermined number of medical records with the largest Shape values ​​are identified as one or more of the above-mentioned medical records.

[0073] Note that the Shap value is an example of an external evaluation criterion calculated for medical records. Therefore, in example process 150, a different external evaluation criterion may be used instead of the Shap value. Examples of other external evaluation criteria include the Gain value, Cover value, and Weight value. That is, step 165 is an example of a step that identifies one or more medical records based on an external evaluation criterion.

[0074] The methods described above are merely examples, and the methods for identifying one or more medical records that contribute to the model's classification performance are not limited to these.

[0075] 4. Use of the generated outcome definitions Figure 4 is a flowchart of Exemplary Method 400 for using an outcome definition for a given outcome. Exemplary Method 400 assumes that the generated outcome definition does not use a certain clinical test value. Furthermore, Exemplary Method 400 may be executed by a computer, and therefore, each step included in Exemplary Method 400 may also be executed by a computer.

[0076] Paragraph 410 shows a step for generating an outcome definition for a certain outcome that does not use the above-mentioned clinical test value. This outcome definition may be generated by example method 100, but is not limited to this.

[0077] Procedure 420 shows the step of retrieving one or more medical records of a patient from a second database. The second database is a database that does not include the aforementioned clinical test values ​​as medical records, at least for the patient in question.

[0078] 430 describes a step of determining whether a certain outcome has occurred in the patient, based on one or more of the medical records obtained, according to the generated outcome definition.

[0079] 440 indicates a step of adding information to the second database that shows whether the patient has exhibited the aforementioned outcome.

[0080] Furthermore, the information indicating whether the above-mentioned outcome has occurred in the patient may be information about the above-mentioned clinical test value used to determine whether the above-mentioned outcome has occurred. For example, if the above-mentioned outcome is "having developed diabetic nephropathy," then according to a known outcome definition that uses the clinical test value "eGFR" (not the outcome definition generated by example method 100), if the patient has developed "diabetic nephropathy," then the patient's "eGFR" will be less than 30, and if the patient has not developed "diabetic nephropathy," then the patient's "eGFR" will be 30 or higher. Therefore, in this case, the information indicating whether the above-mentioned outcome has occurred in the patient may be, when the above-mentioned outcome ("having developed diabetic nephropathy") has occurred in the patient, information indicating that the patient has developed diabetic nephropathy or that the eGFR is less than 30, and when the above-mentioned outcome ("having developed diabetic nephropathy") has not occurred in the patient, information indicating that the patient does not have diabetic nephropathy or that the eGFR is 30 or higher.

[0081] According to Exemplary Method 400, information indicating whether a patient exhibits the aforementioned outcome can be added to a database that does not currently contain such information. In other words, according to Exemplary Method 400, a new database can be generated to which information indicating whether a patient exhibits the aforementioned outcome has been added.

[0082] The technical significance of example method 400 will be explained in more detail below.

[0083] According to a survey by the Japanese Society for Pharmacoepidemiology, 24 medical information databases have been identified that can be used for clinical research, but only half of these databases offer clinical laboratory values. Specifically, of the 24 databases, 12 are based on data from hospitals and other medical institutions (medical institution-based), 8 are based on data from insurers such as national health insurance (insurer-based), and the remaining 4 are based on data from pharmacies and other insurance-based pharmacies (insurance-pharmacy-based). Among these, only the medical institution-based databases contain clinical laboratory values. On the other hand, there are significant differences in age groups and the number of cases included in medical information databases, so it is necessary to appropriately select the medical information database to be used according to the purpose of the clinical research. In other words, depending on the purpose of the clinical research, it may be necessary to use a database that offers clinical laboratory values ​​even if it is desired. For example, MDV provides both medical institution-based and insurer-based medical information databases. The former includes clinical laboratory values ​​for many cases and some cases, but both have few cases of elderly people, so these medical information databases are not selected for clinical research targeting elderly populations. In clinical research, particularly targeting the elderly, insurer-based medical information databases that utilize data from the late-stage elderly healthcare system are often employed.

[0084] On the other hand, according to example method 400, even if it is necessary to select a medical information database that does not include clinical test values ​​from a clinical research perspective, the database can be used after adding information about clinical test values.

[0085] 5. System for generating outcome definitions Figure 5 is a diagram illustrating the configuration of an exemplary system 500 for generating an outcome definition for a certain outcome, according to one embodiment of the present disclosure.

[0086] 510 represents one or more computers. One or more computers 510 as a whole are configured to perform one or both of Exemplary Method 100 and Exemplary Method 400. Some or all of the one or more computers 510 may be physical computers, virtual computers, or computers on a so-called cloud.

[0087] 520 indicates a network that includes either or both a local network and / or the Internet, which are operationally connected to one or more computers 510.

[0088] 530 indicates one or more first databases operationally connected to network 520. Each of the one or more first databases 530 may be one of the first databases described above with respect to the exemplary process 200.

[0089] 540 indicates one or more second databases operationally connected to network 520. Each of the one or more second databases 540 may be one of the second databases described above with respect to the exemplary method 400.

[0090] In Figure 5, one or more first databases 530 and one or more second databases are described as separate entities. However, at least a portion of one or more first databases 530 may be at least a portion of one or more second databases 540. That is, as described above, the first database is a database that includes the above-mentioned clinical test values ​​as medical records for one or more patients, but it is not necessary to include the above-mentioned clinical test values ​​for all patients recorded in the database. In other words, the portion of the medical records of patients that do not include the above-mentioned clinical test values ​​recorded in the first database can be treated as the second database.

[0091] Furthermore, one or more computers 510 configured to perform Exemplary Method 100 may not use one or more first databases 530, for example, if step 120 of Exemplary Method 100 includes a step of reading a given set of training data from the computer's auxiliary storage device or an external storage device into the computer's main memory. Also, if one or more computers 510 are not configured to perform Exemplary Method 400, one or more second databases 540 will not be used. Therefore, it will be understood that the network 520, one or more first databases 530, and one or more second databases 540 are not essential in the Exemplary System 500.

[0092] One or more computers 510 configured to execute one or both of Exemplary Method 100 and Exemplary Method 400 may be implemented using a program. That is, a program may cause one or more computers 510 to execute one or both of Exemplary Methods 100 and 400.

[0093] 6. Examples of Outcome Definition Generation The following describes an example of generating outcome definitions. In this example, one of the outcomes is developing diabetic nephropathy.

[0094] According to the inventors, based on medical records from July 2008 to July 2021 in RWD's database, 22,936 training data were prepared for 22,936 patients aged 18 or older in the same month, three months or more after the initial medical record, where both a diagnosis of type 2 diabetes and a new exposure to diabetes treatment were recorded (step 110 of Exemplary Method 100). The initial medical record for a given patient is the medical record with the earliest recording date among all medical records for that patient in the database (all medical records from July 2008 to July 2021).

[0095] Of the 22,936 patients, half (11,468 patients) had a clinical record of an eGFR of less than 30, and are hereinafter referred to as the "case group." The other half of the 22,936 patients (11,468 patients) were those who did not have records of dialysis, kidney transplantation, diabetic nephropathy diagnosis, or a clinical record of an eGFR of less than 30, but had a clinical record of an eGFR of 30 or higher, and are hereinafter referred to as the "control group." Specifically, for the case group, training data was prepared based on medical records for one month before and after the first diagnosis month in which a clinical record of an eGFR of less than 30 was obtained. For the control group, training data was prepared based on medical records for one month before and after the first diagnosis month in which a clinical record of an eGFR of 30 or higher was obtained.

[0096] The "data encoding one or more medical records of a patient" in this case had 5,631 features. That is, the number of variables or vector elements corresponding to the "data encoding one or more medical records of a patient" was 5,631. More specifically, considering the verification using MDV's database described later, 14,797 medical records (features) common to both RWD's database and MDV's database were extracted, and then features corresponding to medical records with a predetermined number or less in the database were deleted, resulting in 5,631 features. Of the 14,797 features, 2 corresponded to patient background information records, 5,686 to diagnostic records, 6,495 to drug administration records, and 2,614 to medical treatment records.

[0097] Half of the case group (5,734 individuals) had their training data used, and the other half (5,734 individuals) had their validation data used. Similarly, half of the control group (5,734 individuals) had their training data used, and the other half had their validation data used.

[0098] Using a total of 11,468 training data points, a machine learning model called XGBoost was generated to infer whether a patient has diabetic nephropathy. Based on the multiple decision trees included in the model, one or more medical records that contribute to the classification performance of the generated model regarding whether a certain outcome occurs in a particular patient were identified (step 120 of example method 100). As a result, 1,096 medical records were identified.

[0099] Based on these 1,096 identified medical records, outcome definitions were generated (step 140 of Exemplary Method 100). The positive predictive value (PPV) and sensitivity for diabetic nephropathy in the validation data were calculated according to these outcome definitions, yielding 0.8558 and 0.8279, respectively. Similarly, the negative predictive value (NPV) and specificity for diabetic nephropathy in the validation data were calculated according to these outcome definitions, yielding 0.8333 and 0.8605, respectively. For reference, the values ​​for the training data according to these outcome definitions were: positive predictive value: 0.9953, sensitivity: 0.9672, negative predictive value: 0.9681, and specificity: 0.9955. Furthermore, when calculating the positive predictive value and sensitivity for diabetic nephropathy according to the generated outcome definition for data obtained from MDV's database (a separate database from RWD's), the results were 0.8053 and 0.7260, respectively. Similarly, when calculating the negative predictive value and specificity for diabetic nephropathy according to the generated outcome definition for data obtained from MDV's database, the results were 0.7505 and 0.8244, respectively. With positive predictive value and sensitivity at this level, it can be considered that the generated outcome definition is valid. In particular, it should be noted that, with the outcome definition generated by the technology disclosed in this disclosure, even though negative predictive value and specificity tend to decrease when positive predictive value and PPV are high, all of the positive predictive value, sensitivity, negative predictive value, and specificity are maintained at high values, ensuring high classification performance.

[0100] 7. About Computers The following describes an example of a computer hardware configuration that can be used to implement one embodiment of this disclosure. The computer that can be used to implement one embodiment of this disclosure may be any computer, for example, a personal computer or a so-called cloud computer, but is not limited thereto. The computer that can be used to implement one embodiment of this disclosure may be a GPU, or may include a GPU.

[0101] Figure 6 shows an example of a computer hardware configuration. As shown in the figure, the computer 600 mainly consists of a processor 610, a main memory 620, an auxiliary memory 630, an input / output interface 640, and a communication interface 650 as hardware resources, which are interconnected via a bus line 660 that includes an address bus, a data bus, a control bus, etc. Interface circuits (not shown) may be interposed between the bus line 660 and each hardware resource as appropriate.

[0102] The processor 610 controls the entire computer, or at least a part of it, including the CPU and microprocessors. A single computer may contain multiple processors 610. In such cases, the term "processor" in the above description may refer collectively to multiple processors 610.

[0103] The main memory 620 is a volatile memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory) that provides a working area to the processor 610.

[0104] The auxiliary storage device 630 is a non-volatile memory such as an HDD, SSD, or flash memory that stores software programs and data. These programs and data are loaded from the auxiliary storage device 630 to the main memory 620 via the bus line 660 at any given time. The auxiliary storage device 630 may be referred to as a computer-readable storage medium, a non-temporary computer-readable storage medium, or a computer-readable storage medium. A program includes instructions that cause the processor to perform desired processing.

[0105] The input / output interface 640 is used to either present information or receive information input, or both, and includes digital cameras, keyboards, mice, displays, touch panel displays, microphones, speakers, various sensors, etc.

[0106] The communication interface 650 is connected to a network 655, which consists of one or more of the Internet or a local area network (LAN), and sends and receives data via the network 655. The communication interface 650 and the network 655 may be connected by wire or wireless. The communication interface 650 may also acquire network-related information, such as information related to Wi-Fi access points and information related to base stations of communication carriers.

[0107] It will be apparent to those skilled in the art that, through the cooperation of the hardware resources and software exemplified above, the computer 600 can function as a desired means, perform the desired steps, and realize the desired functions.

[0108] 8. Conclusion While several examples of embodiments of this disclosure have been described above, it should be understood that these are merely illustrative and do not limit the technical scope of this disclosure. It should be understood that modifications, additions, and improvements to the embodiments can be made as appropriate without departing from the spirit and scope of this disclosure. The technical scope of this disclosure should not be limited by any of the embodiments described above, but should be defined solely by the claims and their equivalents. [Explanation of Symbols]

[0109] 100... An example method for generating an outcome definition for a given outcome. 140…Example decision tree 141...Data (Training Data) 142... Branch 143... Branching conditions 144…Output 150...Example processes that step 120 may include Example process for preparing multiple training data sets (200, 250, etc.) 300…Example graph where the vertical axis represents medical records and the horizontal axis corresponds to their Shape values. 310... Prescribed value 320...One or more identified medical records 400...Examples of using outcome definitions for a given outcome. 500... An exemplary system for generating outcome definitions for a given outcome. 520, 655… Network

Claims

1. A method for generating an outcome definition for a certain outcome, The step involves preparing multiple training data sets, where each training data set is: Data encoded from one or more medical records of the patient, Data encoded to indicate whether the aforementioned outcome is occurring in the patient, and Steps including, The step of generating a model that infers whether a certain outcome occurs in a patient by machine learning using the aforementioned multiple training data, and identifying one or more medical records that contribute to the classification performance of whether a certain outcome occurs in a patient, wherein the model is Encoded data of the medical records of the aforementioned patient At least the following are inputs: Data encoding whether the aforementioned outcome occurs in the aforementioned patient. A step that produces at least the following output, The steps include generating the outcome definition based on the identified one or more medical records, and A method that includes this.

2. The method according to claim 1, The one or more medical records, each containing encoded data in the training data, do not include a certain clinical test value, and the outcome definition generated thereby does not use the aforementioned clinical test value. method.

3. The method according to claim 2, wherein the generated outcome definition is: A certain diagnosis was made, The administration of a certain drug, A certain medical procedure was performed and A method that includes at least one of the following.

4. The method according to claim 3, wherein the generated outcome definition is: A clinical test value other than the aforementioned clinical test value is greater than, less than, greater than, or less than a predetermined value. Methods that further include the above.

5. The method according to claim 2, wherein the step of preparing a plurality of training data is: The steps include obtaining one or more medical records of a patient from the first database, which include the aforementioned clinical test values, Based on the acquired medical records, the step of generating data by encoding one or more medical records of the patient that do not include a certain clinical test value, A step of generating data that encodes whether the aforementioned outcome is occurring in the patient, based on one or more medical records obtained, according to a known outcome definition for the aforementioned outcome and a known outcome definition using the aforementioned clinical test value. Methods that include...

6. The method according to claim 1, The aforementioned machine learning method includes gradient boosting, and the resulting model includes multiple decision trees. The one or more medical records identified as contributing to the classification performance of whether a certain outcome is occurring in a particular patient include at least one of the medical records used as branching conditions by the plurality of decision trees. method.

7. The method according to claim 6, wherein the step of preparing a plurality of training data is: A step of calculating a Shap value for each medical record using a predetermined model, different from the generated model, which infers whether a certain outcome occurs in a particular patient, wherein the predetermined model is Encoded data of the medical records of the aforementioned patient At least the following are inputs: Data encoding whether the aforementioned outcome occurs in the aforementioned patient. A step that produces at least the following output, A step of preparing a plurality of training data so that no predetermined training data is included, wherein the predetermined training data is training data that includes data encoding a predetermined medical record, and the predetermined medical record is a medical record whose Shap value is less than or equal to a predetermined value, or is not included in a predetermined number of medical records with the largest Shap values. Methods that include...

8. The method according to claim 1, wherein the step of generating a model for inferring whether a certain outcome occurs in a patient by machine learning using the plurality of training data, and identifying one or more medical records that contribute to the classification performance of whether a certain outcome occurs in a patient, Using the generated model, the steps include calculating external evaluation criteria for each medical record, Based on the aforementioned external evaluation criteria, the step of identifying one or more medical records and Methods that include...

9. The method according to claim 8, The aforementioned external evaluation criterion is the Shap value. The step of identifying one or more medical records based on the aforementioned external evaluation criteria is: The step of selecting one or more medical records whose Shap value is greater than or equal to a predetermined value, or selecting a predetermined number of medical records from the largest Shap values ​​to identify the one or more medical records. including, method.

10. A method for utilizing an outcome definition for a certain outcome that does not use a certain clinical test value, The method according to claim 2 or 5, comprising the step of generating the outcome definition without using a certain clinical test value, The second step involves obtaining one or more medical records of a patient from the second database, The steps include determining whether a certain outcome has occurred in the patient based on the one or more medical records obtained, according to the generated outcome definition, The steps include adding information to the second database indicating whether the aforementioned outcome has occurred in the patient, and A method that includes this.

11. The method according to claim 10, The aforementioned outcome was the development of diabetic nephropathy. The aforementioned clinical test value is eGFR. The information indicating whether the aforementioned outcome has occurred in the patient is, If the aforementioned outcome occurs in the patient, the information indicates that the patient has developed diabetic nephropathy or that the eGFR is less than 30. If the aforementioned outcome does not occur in the patient, it indicates that the patient does not have diabetic nephropathy, or that the eGFR is 30 or higher. method.

12. The method according to claim 1, wherein the medical record is Medical records and, Clinical records and, Diagnostic records and, Drug administration records and, Records of medical procedures, Patient background and Methods that include...

13. A program for generating an outcome definition for a certain outcome, The step involves preparing multiple training data sets, where each training data set is: Data encoded from one or more medical records of the patient, Data encoded to indicate whether the aforementioned outcome is occurring in the patient, and Steps including, The step of generating a model that infers whether a certain outcome occurs in a patient by machine learning using the aforementioned multiple training data, and identifying one or more medical records that contribute to the classification performance of whether a certain outcome occurs in a patient, wherein the model is Encoded data of the medical records of the aforementioned patient At least the following are inputs: Data encoding whether the aforementioned outcome occurs in the aforementioned patient. A step that produces at least the following output, The steps include generating the outcome definition based on the identified one or more medical records, and A program that causes a computer to execute something.

14. A system for generating an outcome definition for a certain outcome, The step involves preparing multiple training data sets, where each training data set is: Data encoded from one or more medical records of the patient, Data encoded to indicate whether the aforementioned outcome is occurring in the patient, and Steps including, The step of generating a model that infers whether a certain outcome occurs in a patient by machine learning using the aforementioned multiple training data, and identifying one or more medical records that contribute to the classification performance of whether a certain outcome occurs in a patient, wherein the model is Encoded data of the medical records of the aforementioned patient At least the following are inputs: Data encoding whether the aforementioned outcome occurs in the aforementioned patient. A step that produces at least the following output, The steps include generating the outcome definition based on the identified one or more medical records, and A system configured to perform [a specific action].

Images