Machine learning models to extract diagnosis, treatment, and key dates

Abstract

A model-assisted system for determining dates of patient events may include a processor that may be programmed to access a database storing medical records related to patients, the medical records including unstructured data, analyze the unstructured data to identify a plurality of pieces of information in the medical records related to the patient events, determine a date associated with each of the plurality of pieces, identify a plurality of query periods related to the patient events, and generate, for each of the query periods, a probability of whether the patient event occurred during the query periods based on the plurality of pieces and the associated dates.

Description

[Technical field]

[0001] background CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 157,369, filed March 5, 2021. The contents of the above application are incorporated herein by reference in their entirety.

[0002] Technical Field

[0002] This disclosure relates to analyzing medical records, and more particularly, to extracting key dates and other information from unstructured medical data. [Background technology]

[0003] Background information

[0003] In today's healthcare system, analyzing patient diagnostic, treatment, test, and other healthcare data across large patient populations can provide useful insights for understanding disease, formulating new forms of therapies and treatments, and evaluating the effectiveness of existing therapies and treatments. In particular, it can be useful to identify specific dates associated with key events or stages during a patient's diagnosis and / or treatment. For example, it can be useful for researchers to identify patients diagnosed with a particular disease and dates associated with the date the disease was diagnosed or a particular stage of the disease. It can be even more useful to extract other dates, such as the date of later advanced diagnosis (e.g., due to recurrence or advanced stage of the disease). This can enable researchers to make decisions across large patient populations, for example, to select patients to include in clinical trials.

[0004]

[0004] Patient information may be contained within an electronic medical record (EMR). However, in many cases, information regarding diagnosis dates or other key events is represented within unstructured data (e.g., doctor's visit notes, laboratory assistant reports, or other text-based data), which may make it difficult to extract relevant date information computationally. For example, a doctor may include notes regarding medical diagnoses in some documents within a patient's medical record without explicitly including the diagnosis date. Thus, determining the exact date of a diagnosis (or similar event) based on an ambiguous note may involve piecing together several pieces of information. Furthermore, the sheer volume of data that researchers must review makes it impossible to manually extract dates or other information. For example, such manual extraction may involve searching thousands, tens of thousands, hundreds of thousands, or millions of patient medical records, each of which may contain hundreds of pages of unstructured text. Thus, it may be very time-consuming and difficult, if not impossible, for a human reviewer to process that amount of data. Thus, extracting key dates from patient medical records using traditional techniques, especially for large patient populations, may quickly become an insurmountable task.

[0005]

[0005] Therefore, in light of these and other deficiencies of current techniques, a technical solution is needed to more accurately extract key dates related to patient diagnoses and treatments. Specifically, the solution should advantageously enable extraction of specific dates (e.g., date of initial diagnosis, date of advanced diagnosis, start date of treatment, end date of treatment, etc.) from unstructured data within a large set of patient EMRs. Summary of the Invention

[0006] overview

[0006] Embodiments consistent with the present disclosure include systems and methods for determining dates of patient events. In one embodiment, a model-aided system may include at least one processor. The processor may be programmed to: access a database storing medical records associated with a patient, the medical records including unstructured data; analyze the unstructured data to identify a plurality of pieces of information in the medical records that are associated with the patient events; determine a date associated with each of the plurality of pieces; identify a plurality of query periods associated with the patient events; and generate, for each of the query periods, a probability of whether the patient event occurred during the query period based on the plurality of pieces and the associated dates.

[0007] In one embodiment, a method for determining dates of a patient event is disclosed that may include accessing a database that stores medical records related to a patient, the medical records including unstructured data, parsing the unstructured data to identify a plurality of pieces of information in the medical records related to the patient event, determining a date associated with each of the plurality of pieces, identifying a plurality of query periods related to the patient event, and generating, for each of the query periods, a probability of whether the patient event occurred during the query period based on the plurality of pieces and the associated dates.

[0008]

[0008] Consistent with other disclosed embodiments, a non-transitory computer-readable storage medium may store program instructions that, when executed by at least one processor, perform any of the methods described herein.

[0009] BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, and together with the description, serve to illustrate the principles of various exemplary embodiments. [Brief description of the drawings]

[0010] [Figure 1]

[0010] FIG. 1 is a block diagram illustrating an exemplary system environment for implementing embodiments consistent with this disclosure. [Diagram 2]

[0011] FIG. 1 is a block diagram illustrating an example medical record for a patient, consistent with disclosed embodiments. [Diagram 3]

[0012] 1 illustrates an example process for extracting snippets of text within unstructured patient medical record data consistent with disclosed embodiments. [Figure 4]

[0013] 1 shows an example of a set of documents that may be analyzed to determine dates associated with a patient, consistent with disclosed embodiments. [Diagram 5]

[0014] FIG. 1 is a schematic diagram of an example of a trained model and inputs and outputs to the model, consistent with disclosed embodiments. [Figure 6]

[0015] FIG. 2 is a schematic diagram of an example process for determining a query output for a query date, consistent with disclosed embodiments. [Figure 7]

[0016] FIG. 13 is a schematic diagram of an example process for generating probabilities based on query output vectors, consistent with disclosed embodiments. [Figure 8]

[0017] 1 is a flow diagram of an example process for extracting patient information, consistent with disclosed embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Detailed Description

[0018] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. Although several exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, the illustrated components can be substituted, added, or modified, and the exemplary methods described herein can be modified by substituting, rearranging, removing, or adding steps to the disclosed methods. Thus, the following detailed description is not limited to the disclosed embodiments and examples. Rather, the appropriate scope is defined by the appended claims.

[0012]

[0019] The embodiments disclosed herein include computer-implemented methods, tangible non-transitory computer-readable media, and systems. The computer-implemented methods may be executed by at least one processor (e.g., processing device) that receives instructions, for example, from a non-transitory computer-readable storage medium. Similarly, a system consistent with the present disclosure may include at least one processor (e.g., processing device) and a memory, where the memory may be a non-transitory computer-readable storage medium. As used herein, a non-transitory computer-readable storage medium refers to any type of physical memory in which information or data readable by at least one processor may be stored. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage medium. Singular terms such as "memory" and "computer-readable storage medium" may further refer to multiple structures, such as multiple memories and / or computer-readable storage media. As referred to herein, "memory" may include any type of computer-readable storage medium unless otherwise specified. A computer-readable storage medium can store instructions for execution by at least one processor, including instructions for causing the processor to perform steps or stages consistent with the embodiments herein. In addition, one or more computer-readable storage media can be utilized in implementing a computer-implemented method. The term "computer-readable storage medium" should be understood to include tangible items and to exclude carrier waves and transient signals.

[0013]

[0020] The disclosed systems and methods can automate the analysis of medical records of patients or patient populations to identify dates associated with key events during the patient's diagnosis and treatment. For example, researchers, physicians, clinicians, or other users may be interested in identifying patients who have been diagnosed with a particular disease and the estimated date of diagnosis, or patients who are taking a particular medication and the dates associated with such treatment. This can enable users to efficiently make various decisions regarding individual patients within a large population based on analysis of the EMR. For example, researchers can identify patients who have been diagnosed with an advanced stage of disease and the date of advanced diagnosis, which information can indicate whether the patient may be a candidate for inclusion in a clinical trial or other type of cohort.

[0014]

[0021] Figure 1 illustrates an example of a system environment 100 for implementing embodiments consistent with the present disclosure, which are described in detail below. As shown in Figure 1, system environment 100 may include several components, including client device 110, data source 120, system 130, and network 140. It will be understood from this disclosure that the number and arrangement of these components are exemplary and are shown for illustrative purposes. Other arrangements and numbers of components may be used without departing from the teachings and embodiments of the present disclosure.

[0015]

[0022] As shown in FIG. 1, the exemplary system environment 100 may include a system 130. The system 130 may include one or more server systems, databases, and / or computing systems configured to receive information from entities over a network, process the information, store the information, and display / transmit the information to other entities over the network. Thus, in some embodiments, the network may facilitate cloud sharing, storage, and / or computing. In one embodiment, the system 130 may include a processing engine 131 and one or more databases 132, which are illustrated within the area bounded by the dashed lines representing the system 130. The processing engine 131 may include at least one processing device, such as one or more general-purpose processors, e.g., a central processing unit (CPU), a graphics processing unit (GPU), etc., and / or one or more special-purpose processors, e.g., an application specific integrated circuit (ASIC), field programmable gate array (FPGA), etc.

[0016]

[0023] The various components of the system environment 100 may include hardware, software, and / or firmware assemblies including memory, a central processing unit (CPU), and / or a user interface. The memory may include any type of RAM or ROM implemented by a physical storage medium, such as magnetic storage including floppy disks, hard disks, or magnetic tapes, semiconductor storage such as solid state disks (SSD) or flash memory, optical disk storage, or magneto-optical disk storage. The CPU may include one or more processors for processing data according to a set of programmable instructions or software stored in the memory. The functions of each processor may be provided by a single dedicated processor or by multiple processors. Additionally, the processor may include, without limitation, digital signal processor (DSP) hardware, or any other hardware capable of executing software. The optional user interface may include any type or combination of input / output devices, such as a display monitor, keyboard, and / or mouse. A user of the system environment 100 may include any individual who may wish to access and / or analyze patient data. Thus, throughout this disclosure, references to a "user" of the disclosed embodiments may encompass any individual, such as a physician, a researcher, a quality assurance department at a healthcare institution, and / or any other individual.

[0017]

[0024] Data transmitted and / or exchanged within system environment 100 may occur over a data interface. As used herein, a data interface may include any boundary across which two or more components of system environment 100 exchange data. For example, environment 100 may exchange data between software, hardware, databases, devices, humans, or any combination of the above. Furthermore, it will be understood that any suitable configuration of software, processors, data storage, and networks may be selected to implement the components of system environment 100 and features of related embodiments.

[0018]

[0025] The components of environment 100 (including systems 130, client devices 110, and data sources 120) can communicate with each other and with other components via network 140. Network 140 may include various types of networks, such as the Internet, a wired wide area network (WAN), a wired local area network (LAN), a wireless WAN (e.g., WiMAX), a wireless LAN (e.g., IEEE 802.11, etc.), a mesh network, a mobile / cellular network, an enterprise or private data network, a storage area network, a virtual private network using a public network, a short-range wireless communication technique (e.g., Bluetooth, infrared, etc.), or various other types of network communications. In some embodiments, communications may occur across two or more of these types of networks and protocols.

[0019]

[0026] The system 130 may be configured to receive and store data transmitted over the network 140 from various data sources, including the data source 120, process the received data, and transmit data and results based on the processing to the client device 110. For example, the system 130 may be configured to receive patient data from the data source 120 or other sources in the network 140. In some embodiments, the patient data may include medical information stored in the form of one or more medical records. Each medical record may be associated with a particular patient. The data source 120 may relate to a wide variety of sources of medical information regarding a patient. For example, the data source 120 may include a patient's health care provider, such as a doctor, nurse, consultant, hospital, clinic, etc. The data source 120 may also relate to a laboratory, such as a radiology or other imaging laboratory, a hematology laboratory, a pathology laboratory, etc. The data source 120 may also relate to an insurance company or any other source of patient data.

[0020]

[0027] The system 130 may further communicate with one or more client devices 110 via the network 140. For example, the system 130 may provide the client device 110 with results based on an analysis of information from the data source 120. The client device 110 may include any entity or device capable of receiving or transmitting data via the network 140. For example, the client device 110 may include a computing device such as a server or a desktop or laptop computer. The client device 110 may also include other devices such as mobile devices, tablets, wearable devices (i.e., smart watches, embedded devices, fitness trackers, etc.), virtual machines, IoT devices, or various other technologies. In some embodiments, the client device 110 may transmit queries to the system 130 via the network 140 for information about one or more patients, such as patients having or associated with a particular attribute, dates when the patient is associated with the attribute, dates of events associated with the patient, or various other information about the patient.

[0021]

[0028] In some embodiments, the system 130 may be configured to analyze patient medical records (or other forms of unstructured data) to determine dates associated with key events during the patient's diagnosis and / or treatment. For example, the system 130 may analyze the patient's medical records to determine the date the patient was diagnosed with a particular condition (e.g., metastases in a particular body region), the date of a test for a particular condition, the date the patient tested positive or negative for a particular condition, the start or end date of a particular treatment or type of treatment (e.g., taking a particular medication), the date of an operation (e.g., surgery), or various other dates. As described further below, the system 130 may be configured to use one or more machine learning models to perform this analysis. Although patient medical records are used as an illustrative example throughout this disclosure, it will be understood that in some embodiments the disclosed systems, methods, and / or techniques may be used to identify patients exhibiting a condition from other types of records as well.

[0022]

[0029] To efficiently extract a particular disease or treatment along with key dates and stages associated with the disease or treatment, the system 130 may be configured to access a database that stores medical records associated with one or more patients. A medical record may refer to any type of document that includes data related to a patient's diagnosis and / or treatment. In some embodiments, a patient may be associated with multiple medical records. For example, a doctor associated with the patient, a nurse associated with the patient, a physical therapist associated with the patient, a pathologist, a radiologist, etc. may each generate a medical record for the patient.

[0023]

[0030] 2 is a block diagram illustrating an example medical record 200 for a patient consistent with disclosed embodiments. The medical record 200 may be received from a data source 120 and processed by the system 130 to identify dates associated with the patient, as described above. As shown in FIG. 2, the record received from the data source 120 (or elsewhere) may include one or both of unstructured data 210 and structured data 220. The structured data 220 may include quantifiable or categorizable data about the patient, such as gender, age, race, weight, vital signs, lab results, date of diagnosis, type of diagnosis, staging (e.g., billing codes), timing of treatment, procedures performed, dates of visits, type of clinic, insurance provider and start date, medication instructions, administration of medication, or any other measurable data about the patient.

[0024]

[0031] As explained above, much of the information relevant to making decisions about a patient, such as dates of treatment or diagnosis, may be stored in unstructured data in a patient medical record. As used herein, unstructured data may include information about a patient that is not quantifiable or easily categorized, such as a doctor's notes or patient test reports. For example, unstructured data 210 may include information such as a doctor's description of a treatment plan, notes describing what happened during a medical visit, a patient's statements or stories, a subjective assessment or description of the patient's health, radiology reports, pathology reports, lab reports, or any other type of information that is not stored in a structured format.

[0025]

[0032] Within the data received from the data sources 120, each patient may be represented by one or more records generated by or for one or more healthcare professionals. For example, a doctor associated with the patient, a nurse associated with the patient, a physical therapist associated with the patient, etc. may each generate a medical record for the patient. In some embodiments, one or more records may be collated and / or stored in the same database. In other embodiments, one or more records may be distributed across multiple databases. In some embodiments, a record may be stored and / or given multiple electronic data representations. For example, a patient record may be represented as one or more electronic files, such as a text file, a Portable Document Format (PDF) file, an Extensible Markup Language (XML) file, etc. If a document is stored as a PDF file, an image, or other file without text, the electronic data representation may also include text associated with the document derived from an optical character recognition process. In some embodiments, unstructured data may be captured by an extraction process, whereas structured data may be entered by a healthcare professional or calculated using an algorithm.

[0026]

[0033] In some embodiments, the unstructured data may include data related to a particular patient's condition. As used herein, a patient's "condition" may refer to any attribute or characteristic related to the patient's well-being or health. The condition may refer to the patient's diagnosed state, such as whether the patient has been diagnosed with a particular disease or illness. In some embodiments, the condition may refer to the stage or status of a particular diagnosed condition. For example, in a patient diagnosed with cancer, the condition may be a particular metastatic site of the cancer (e.g., whether the patient has been diagnosed with metastases in the brain, liver, bone, lungs, adrenal glands, peritoneum, or various other metastatic sites).

[0027]

[0034] The system 130 may be configured to extract dates associated with various patient conditions. For example, such dates may include dates when a condition was developed, observed, tested for, diagnosed, treated, or any other dates associated with a condition. For example, a date may be the start or end date of a particular course of treatment or therapy for a patient. As used herein, a course of treatment (or therapy) may refer to a therapy used to treat a particular disease or condition. For example, a therapy may include administration of a particular drug to a patient (e.g., drug therapy, chemotherapy, etc.), a surgical procedure, gene therapy, immunotherapy, changes in the patient's diet, radiation therapy, physical therapy, counseling or psychotherapy, meditation, sleep therapy, or various other forms of treatment that may be prescribed to a patient. In some embodiments, the dates of interest may vary depending on the particular application, such as the type of cohort selected, the type of study, the type of condition being analyzed, or various other factors.

[0028]

[0035] In many cases, diagnosis information for a particular disease or disease stage or treatment information and key dates for a particular treatment may be represented in unstructured data and may not be clearly tied to a specific date in the patient's medical record. For example, with respect to a patient's diagnosis of metastatic non-small cell lung cancer (NSCLC), a physician may refer to the diagnosis in multiple notes both before and after the diagnosis date. For example, prior to an advanced diagnosis, the physician may include notes such as "presenting with symptoms of NSCLC, no evidence of metastasis" and "possible metastasis to liver". Various documents following an advanced diagnosis may include phrases such as "biopsy shows metastasis to liver" and "patient with metastatic NSCLC". Thus, using conventional techniques, it may be difficult for an automated system to ascertain the exact date of an advanced diagnosis or disease stage based on the notes. Similar issues may arise with treatment information related to a disease. For example, prior to initiating a particular treatment for a disease, a physician may include a note indicating the diagnosis of the disease and / or a note indicating that various treatment options were discussed with the patient. After treatment is initiated, documents in the patient's record may indicate a response to treatment but may not indicate the specific date that treatment was initiated. Thus, extracting accurate dates of treatment can also be difficult using conventional techniques.

[0029]

[0036] To overcome these and other difficulties, the system 130 can extract fragments of text related to events of interest. As discussed above, the system 130 can be configured to determine dates associated with a particular event or condition. Following the example of a patient's diagnosis date for metastatic non-small cell lung cancer above, such fragments may include the fragments "presenting with symptoms of NSCLC, no evidence of metastasis," "possible metastasis to liver," "biopsy showed metastasis to liver," and "patient with metastatic NSCLC." In some embodiments, the system 130 can perform a search against a set of documents (which may be a set of patient medical records) for keywords related to a particular event or condition. For example, the system 130 can perform a search 510 against one or more unstructured medical record documents to extract fragments related to the diagnosis date for the patient's condition. In the case of a diagnosis of NSCLC, the relevant terms may include "NSCLC," "lung cancer," "metastasizing," "metastasis," "metastatic," "spread," or various other words, symbols, acronyms, or phrases that may be related to a particular event. These sentences or fragments can be tokenized and represented as a series of tokenized vectors. These tokenized vectors can be processed to generate corresponding vectorized sentences.

[0030]

[0037] FIG. 3 illustrates an example of a process 300 for extracting fragments of text in unstructured data of a patient medical record consistent with disclosed embodiments. In step 310, the system 130 can perform a search against a set of documents for keywords related to a particular event or condition. In this example, the date of interest may be the date of diagnosis for metastasis to the brain. Thus, the keywords may include "brain," "temporal," "occipital," "frontal," or other terms that may be generally related to the brain. Thus, the system 130 can identify terms 322 in the unstructured text as shown in step 320. The system 130 can then extract fragments of text around the relevant terms in the unstructured data. For example, fragments 332 around the term 322 can be extracted from the unstructured text as shown in step 330. The length or structure of the fragments can be specified in various ways. In some embodiments, the fragments 332 can be defined based on a predefined window. For example, the fragment may be defined based on a predetermined number of characters (e.g., 20 characters, 50 characters, 60 characters, or any suitable number of characters to capture the context of the use of the term) before and after the target term 322 in the text. The window may also be defined to take into account word boundaries, such as by widening or narrowing the window (e.g., based on punctuation, etc.) to end at a word boundary or sentence boundary, so that partial words are not included at the ends of the fragment. In some embodiments, the window may be defined based on a predefined number of words or other variables.

[0031]

[0038] These sentences or fragments can be tokenized and represented as a series of tokenized vectors 340. For example, the system 130 can replace the term 322 with the tokenized term. This can ensure that the patient's condition is expressed using the same terminology in each of the extracted fragments. For example, a document containing "brain" and a document containing "cerebrum" can both result in extracted fragments containing the term "_brain_" or a similar token. The use of tokens can also improve the performance of the machine learning model by reducing feature sparsity, accelerating training time, and allowing the model to converge with a more limited set of labeled data. The same tokenization process can be performed for other words in the fragment, such that each word is represented by a token. Each fragment can thus be represented as a vector of values, each of which is a tokenized representation of a word contained in the fragment. In some embodiments, the fragment vector can have a predetermined size such that the extracted fragments have a uniform size. As a result, a tokenized fragment vector 340 can be extracted from the document. In some embodiments, this can include applying a gated recurrent unit (GRU) network followed by an attention layer and a feedforward layer. An example process for generating these fragment vectors is described in commonly assigned U.S. Patent Application Publication No. 2021 / 0027894 A1 and WO 2020 / 092316, the contents of which are incorporated herein by reference in their entireties. The resulting fragment vectors can be input as input vectors into a trained machine learning model.

[0032]

[0039] The system may further associate each input vector (i.e., vectorized sentence) with a date. In some embodiments, such a date may include a date associated with the document from which the snippet was extracted. Alternatively or additionally, various other dates may be associated with each input vector. For example, if text within or around the snippet contains a particular date, that date may be used instead of the document date. FIG. 4 illustrates an example of a set of documents that may be analyzed to determine a date associated with a patient, consistent with disclosed embodiments. In this example, a set of documents 410, 420, 430, 440, and 450 may be analyzed to determine the date the patient was diagnosed with or showed symptoms of liver metastasis. For illustrative purposes, documents 410, 420, 430, 440, and 450 are shown along a timeline 400 that corresponds to the document date of each document. Each of documents 410, 420, 430, 440, and 450 may be associated with a document date. The document date may refer to the date the document was created and may be extracted from metadata or other data associated with the document. The document date may refer to various other dates related to the document, such as the date the document was updated, the revision date, the receipt date, the publication date, or any other relevant date. For example, document 410 may relate to May 1, 2016, and thus the fragments extracted from document 410 may relate to this date.

[0033]

[0040] In some embodiments, various other dates can be associated with the fragment. For example, if the text in or around the fragment contains a specific date, that date can be used instead of the document date. As shown in FIG. 4, document 450 can contain the fragment "Transferred from February 15, 2017." The document 450 is dated March 13, 2018, but a February 2017 date may be more indicative of the date of interest. Thus, the fragment can be associated with a February 2017 date instead of a March 2018 date. In some embodiments, the document can be parsed for a specific cutoff date 460 (in this case February 14, 2018) and documents associated with dates after the cutoff date can not be considered. However, using a February 2017 date instead of a March 2018 date may allow the fragment of document 450 to be included as an input to the model.

[0034]

[0041] The resulting input vectors and associated input dates can be input to a trained machine learning model configured to determine a particular disease and date associated with the event of interest. FIG. 5 is a schematic diagram of an example of a trained model 540 and inputs and outputs to the model consistent with disclosed embodiments. The trained model 540 can be trained to receive as input a set of tokenized fragment vectors 510 and paired dates 520 and output a probability 550 indicating the probability that a particular event occurred within a particular date range. In some embodiments, the trained model 540 can include a feed-forward network. A variety of machine learning algorithms can be used, including neural networks, logistic regression, linear regression, regression, random forests, K-nearest neighbor (KNN) models (e.g., as described above), K-means models, decision trees, Cox proportional hazards regression models, naive Bayes models, support vector machine (SVM) models, gradient boosting algorithms, or any other type of machine learning model or algorithm.

[0035]

[0042] The fragment vector 510 may be represented as vectorized fragments extracted from the unstructured data as described above with respect to FIG. 3. For example, the fragment vector 510 may be represented in the form of vectorized fragments 340. The paired dates 520 may represent dates associated with each of the vectorized fragments as described above with respect to FIG. 4. In this example, the fragment vector 512 may be associated with a paired date 522, and another fragment vector 514 may be associated with a paired date 524. The paired dates 520 may be represented in vector form similar to the fragment vector 510. These dates may be represented in a variety of ways. In some embodiments, the representation may include a standardized date format such as YYYY / DD / MM, where "YYYY" represents the year value, "MM" represents the month value, and "DD" represents the day value. In some embodiments, the dates may be represented as a number of days away from or relative to a reference date, as shown in FIG. 3. For example, the date may be expressed as the number of days before or after the censoring date (e.g., date 460 above), a date relative to a cohort (e.g., the censoring date for a particular diagnosis), the current date, or any other date that may be used as a reference point.

[0036]

[0043] In some embodiments, one or more query dates 530 may be input to the model in addition to the vector of fragments 510 and paired dates 520. Each query date may be a point in time and represent a time point for which the trained model 540 should make a prediction. In some embodiments, the query may be a series of evenly spaced dates. For example, the query may be a series of dates spaced 7 days apart. In some embodiments, the query dates 530 may be automatically generated to encompass the paired dates 520. Alternatively or in addition, the query dates 530 may depend at least in part on user input. For example, the query dates 530 may each be manually defined based on input from a user. As another example, a user may input an interval for the query dates 530, and the system 130 may generate the query dates 530 to encompass the paired dates 520 with the interval defined by the user. In some embodiments, this may include presenting one or more elements in a user interface (e.g., by the client device 110) and receiving user input via one or more elements of the user interface. A weekly query is used as an example, but various other time periods may be used. For example, the period may be daily, several days, biweekly, monthly, yearly, or any other suitable period.

[0037]

[0044] For each of the query dates 530, the trained model 540 can generate a prediction of whether the date of interest occurred within a range or time period relative to the query date. For example, such ranges may include a range encompassing the query date (e.g., within 1 day, within 5 days, within 10 days, within 30 days, or any other suitable range of the query date), a range before the query date, and a range after the query date. Thus, the trained model 540 may output a number of probabilities associated with each of the query dates 530 based on the vector of fragments 510 and the paired dates 520. For example, the query dates 530 may include a query date 532, which in this example may be a date 14 days before the reference date. The trained model 540 may output a set of probabilities 552 indicating whether the date of interest occurred before the query date 532, during the query date 532 (or within the range of the query date 532, such as within 30 days of the query date 532), or after the query date 532. In the illustrated example, the probabilities 552 may include a 90% probability that the date of interest occurred before the query date 532 (or a range encompassing the query date 532), a 9% probability that the date of interest occurred during the query date 532 (or a range encompassing the query date 532), and a 1% probability that the date of interest occurred after the query date 532 (or a range encompassing the query date 532). The model can then generate a distribution of probabilities over a set of query dates, with each query returning a probability of whether the date of interest occurred during that query.

[0038]

[0045] In some embodiments, the fragment vector 510 may be further processed before being input to the trained model 540. For example, to determine the probability of a given query (e.g., to determine the probability 552 of a query date 532), the system 130 may analyze the fragment vector 510 within several time windows for the query. In some embodiments, such analysis may include applying one or more aggregation functions to the associated fragments for each window. FIG. 6 is a schematic diagram of an example of a process 600 for determining a query output for a query date 532, consistent with disclosed embodiments. In the example shown in FIG. 6, one or more query outputs 640 may be generated based on the query date 532. Such generation may include evaluating the fragment vector 510 against a series of time windows 610 as shown. A time window may be a range of time relative to a reference date 612 (corresponding to the query date 532). For example, as shown in Figure 6, the time window may be 365 days or more before the query date, between 365 and 30 days before the query date, between 30 and 7 days before the query date, less than 7 days before the query date, within 7 days after the query date, between 7 and 30 days after the query date, between 30 and 365 days after the query date, and 365 days or more after the query date. These time windows are provided by way of example, and any other suitable time windows may be used.

[0039]

[0046] For each time window, the vector of snippets associated with the input dates included in the time window can be analyzed according to one or more aggregation functions. For example, the aggregation functions can include a sum function 620, a mean function 622, and a LogSumExp function 624. The Sum function 620 can represent a vector sum of the input vectors associated with the dates in the time window. For example, a matrix M can be generated as a vector of all of the input vectors for the model. A logical matrix D can be generated with elements indicating whether the corresponding input vector is included in the time window, such that D * Multiplication by M results in the sum of the associated vectors.

[0040]

[0047] In the example shown in FIG. 6, sum function 620, mean function 622, and LogSumExp function 624 are performed over time window 614. In this example, time window 614 represents a time window that spans from 7 days after query date 532 to 30 days after query date 532. Thus, when applied over time window 614, sum function 620 is the sum of any fragment vectors associated with dates that span from 7 days after query date 532 to 30 days after query date 532. In this example, query date 532 is represented as a 14-day date relative to a reference date. Thus, time window 614 ranges from 21 days relative to the reference date to 44 days relative to the reference date. Referring to the example of fragment vector 510 and paired dates 520 shown in FIG. 5, paired dates 522 and 524 are included in the range defined by time window 614, such range includes the sum of fragment vectors 512 and 514. As shown in FIG. 6, applying a sum function 620 within a time window 614 involves generating a sum of the fragment vectors 512 and 514 that may result in a query output 630.

[0041]

[0048] Similarly, the mean function 622 may represent the average value of the input vectors associated with a time window. For example, the logical matrix D may be divided by its sum along the second dimension, so that D * The multiplication by M results in the average of the associated vectors. Although not shown in FIG. 6 for simplicity, a mean function 622 is applied to the fragment vectors 512 and 514 to generate the corresponding query output for the time window 614. The LogSumExp function 624 may be a smooth approximation to a maximum function defined as the logarithm of the sum of the exponentials of its arguments (i.e., the “RealSoftMax” or “TrueSoftMax” function). For example, the torch.exp() function can be applied to the matrix M, and the resulting matrix can be denoted as (D * The resulting output can be multiplied by a logical matrix D (as M). The torch.log() function can be applied. A LogSumExp function 624 is also applied to the fragment vectors 512 and 514 to generate the corresponding query output for the time window 614.

[0042]

[0049] A sum function 620, a mean function 622, and a LogSumExp function 624 are each applied for each time window 610. As a result, multiple aggregations for the input vector associated with the time window may be generated for each time window. This can be repeated for each time window of a given query to generate an output vector for the query. This output vector can be input to a feed-forward network (or a similar form of recurrent neural network) to generate a probability of whether the date of interest occurred within the query date range (or date range). Although sum function 620, mean function 622, and LogSumExp function 624 are shown as examples, one skilled in the art will appreciate that different aggregation functions (or combinations of aggregation functions can be used).

[0043]

[0050] FIG. 7 is a schematic diagram of an example process for generating a probability based on a query output vector, consistent with disclosed embodiments. As shown in FIG. 7, a query output vector 710 may be input to a model (i.e., trained model 540) to generate a probability 552. The query output vector 710 may be generated based on applying a sum function 620, a mean function 622, and a LogSumExp function 624 to the vector of snippets 510 for each of the time windows 610. In other words, for each of the time windows 610, the sum function 620, the mean function 622, and the LogSumExp function 624 may be applied to the vector of snippets associated with the paired dates included in the time window. For example, the query output vector 710 includes, among other query outputs, the query output 532, as shown.

[0044]

[0051] As shown in FIG. 7, the query output vector 710 can be input to the trained model 540. For example, the trained model can be a feed-forward network (or a similar form of recurrent neural network) to generate probabilities 552 for the query date 532. The trained model 540 can thus be trained using a set of training data, which may consist of a set of query output vectors generated based on vectors of fragments and associated dates as described above, along with known dates of interest. This training vector can be input to a training algorithm to generate the trained model 540. The subsequent output vectors can then be input to the trained model 540 to generate probabilities for various query dates. In some embodiments, additional layers can be applied. For example, the trained model 540 can output a score 720, which can be an equal / unequal score of whether the date matches the diagnosis date or other date of interest. These scores can be real-valued and can be converted to probabilities 552 using a Softmax function 730. The various layers shown in FIG. 7 are provided as examples, and one of ordinary skill in the art will recognize other possible layers for manipulating the output of the trained model 540.

[0045]

[0052] The probability 552 may be expressed in a variety of forms. For example, the probability may be expressed as a value within a range (e.g., 0-1, 0-100, etc.), a percentage, a value within a series of graduated values ​​(e.g., 0, 1, 2, 3, etc.), a text-based representation of the probability (e.g., low probability, high probability, etc.), etc. In some embodiments, the model may also output a probability that the date of interest does not occur within the query date, which may be the inverse of the other probability. For example, for a given query date, the model may output a 0.98 probability that the date of interest occurs within the query date and a 0.02 probability that the date of interest does not occur within the query date. The above process may be repeated for each query to generate a probability distribution 550 indicating the probability when the date of interest occurs across a series of query dates. Various other outputs may be generated, such as an overall probability that the patient has been diagnosed with a particular disease, a confidence level associated with the distribution, etc.

[0046]

[0053] In some embodiments, the model may generate probabilities for multiple dates of interest. For example, continuing with the NSCLC diagnosis date example above, the model may output for each query date the probability of whether an initial diagnosis of NSCLC occurred within that query date, and whether an advanced diagnosis (e.g., stage 3b or higher, lower stage with distant metastasis, etc.) occurred within that query date. Thus, for each query output vector, multiple feed-forward layers may be applied such that multiple probabilities are generated.

[0047]

[0054] Although the disclosed systems and methods are generally described using examples of specific diseases and dates associated with diagnoses and various states of the disease, it should be understood that the same or similar processes can be performed for dates of other events. For example, such dates may include start and end dates of a particular treatment or therapy, whether a particular drug was taken along with dosage and date, a particular diagnosis made and associated date, etc. Additionally, various other inputs, such as document type, document format, or other metadata of the document, can also be provided to the trained model.

[0048]

[0055] In some embodiments, the model can be trained by identifying a cohort of patients with a particular disease and identifying various stages of the disease and associated diagnosis dates. For example, the input to the model can include a set of sentences containing keywords related to advanced NSCLC and extracted from the patient's EHR documents. A date can be associated with each sentence using the document's timestamp or a date explicitly mentioned in the sentence, if any. The sentences can be processed by a GRU network and trained to predict the probability of each diagnosis over a series of time points, which can be used to extract whether a patient has been diagnosed with a particular disease and, if so, the date of diagnosis.

[0049]

[0056] In some embodiments, the model can be trained for both disease diagnosis and treatment. For example, the input to the model can include one or more sentences with keywords related to disease diagnosis and sentences with keywords related to disease treatment. The input can also include a training dataset of dates related to disease diagnosis and treatment. For example, the training dataset can include dates when a disease (or various stages of a disease) was diagnosed. Additionally, the training dataset can include dates when a particular course of treatment was initiated as well as other dates related to treatment (e.g., dosage increases, changes in treatment, etc.). The model can be trained using stochastic gradient descent or a similar method for training a model using a set of labeled training data. As a result, the trained model can be configured to extract specific diseases and / or types of treatments as well as dates related to disease diagnosis and treatment.

[0050]

[0057] FIG. 8 is a flow diagram illustrating an example of a process 800 for extracting patient information consistent with disclosed embodiments. Process 800 may be performed by at least one processing device, such as processing engine 131 described above. It should be understood that throughout this disclosure, the term "processor" is used as an abbreviation for "at least one processor." In other words, a processor may include one or more structures that perform logical operations, whether such structures are collocated, connected, or distributed. In some embodiments, a non-transitory computer-readable medium may include instructions that, when executed by a processor, cause the processor to perform process 800. Furthermore, process 800 is not necessarily limited to the steps illustrated in FIG. 8, and any steps or processes of various embodiments described throughout this disclosure, including those described above with respect to FIG. 3, FIG. 4, FIG. 5, FIG. 6, and FIG. 7, may be included within process 800.

[0051]

[0058] At step 810, process 800 includes accessing a database that stores medical records associated with a patient. For example, system 130 can access the patient medical records from local database 132 or from an external data source, such as data source 120. The medical records may include one or more electronic files, such as a text file, an image file, a PDF file, an XLM file, a YAML file, etc. The one or more medical records may correspond to medical record 200 discussed above. In some embodiments, the medical records may include unstructured data, such as unstructured data 210.

[0052]

[0059] In step 820, process 800 includes parsing the unstructured data to identify a plurality of fragments of information in the medical record that are related to a patient event. For example, this step may include identifying fragments 332 as described above. As explained above, a patient event may be any event that may be related to the care of a patient. For example, a patient event may include at least one of a diagnosis date, a treatment date, or an operation date. In some embodiments, process 800 may further include generating a plurality of fragment vectors based on the fragments, such as fragment vector 510.

[0053]

[0060] At step 830, process 800 includes determining a date associated with each of the plurality of fragments. For example, this step may include determining paired dates 520 as described above. In some embodiments, determining a date associated with each of the plurality of fragments may include identifying a date based on metadata of the document in which the fragment is included. For example, as described above with respect to FIG. 4, such dates may include dates when the document was created, saved, published, or various other dates associated with the document. Alternatively or additionally, determining a date associated with each of the plurality of fragments may include identifying a date mentioned within the fragment. For example, the fragment itself may include a date that may be more reflective of a diagnosis date or other event discussed within the fragment.

[0054]

[0061] At step 840, the process 800 includes identifying a plurality of query periods associated with the patient's event. For example, this step may include determining the query date 530 as described above. Identifying the plurality of query periods thus includes identifying a plurality of query dates, the plurality of query periods including at least one period for each of the query dates. For example, the at least one period for each of the query dates may include a period encompassing the query date, a period before the query date, and a period after the query date. The query dates may be identified in various ways. In some embodiments, the plurality of query dates may be dates spaced a week apart, although any other suitable interval may be used. In some embodiments, the query dates, the range of query dates, the interval between query dates, or various other factors may be specified by user input as described above. Identifying the plurality of query periods associated with the patient's event thus may include receiving at least one user input related to the query periods via a user interface. For example, the user interface may be displayed on the client device 110 as described above.

[0055]

[0062] At step 850, the process 800 includes generating a probability for each of the query periods whether a patient event occurred during the query period based on the plurality of fragments and associated dates. For example, this step may include generating the probability 550 as described above. In some embodiments, this step may include various steps described with respect to FIG. 6 and FIG. 7. For example, step 850 may include evaluating the plurality of fragments across a plurality of time windows for each of the plurality of query dates, such as time window 610. In some embodiments, evaluating the plurality of fragments across a plurality of time windows may include processing the fragments associated with the dates included in the time window for each of the plurality of time windows using one or more aggregation functions, as described in more detail above with respect to FIG. 6. For example, the one or more aggregation functions include at least one of a sum function, a mean function, or a LogSumExp function. Step 850 may further include inputting the results of the plurality of functions (i.e., the query output vector 710) into a feedforward network or other type of trained machine learning model.

[0056]

[0063] The above description has been provided for illustrative purposes. It is not exhaustive or limited to the precise form or embodiment disclosed. Modifications and adaptations will become apparent to those skilled in the art from studying the specification and practicing the disclosed embodiments. In addition, although aspects of the disclosed embodiments have been described as being stored in memory, those skilled in the art will appreciate that the aspects can also be stored on other types of computer readable media, such as secondary storage, for example, a hard disk or CD ROM, or other forms of RAM or ROM, USB media, DVD, Blu-ray, 4K Ultra HD Blu-ray, or other optical drive media.

[0057]

[0064] Computer programs based on the descriptions and methods disclosed are within the skill of an experienced developer. The various programs or program modules can be created using any of the techniques known to those skilled in the art, or can be designed in conjunction with existing software. For example, program sections or program modules can be designed in or with the .Net Framework, .Net Compact Framework (and related languages ​​such as Visual Basic, C, etc.), Java, Python, R, C++, Objective-C, HTML, a combination of HTML / AJAX, XML, or HTML with included Java applets.

[0058]

[0065] Moreover, although exemplary embodiments have been described herein, the scope of any and all embodiments with equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations and / or alterations, as will be understood by those skilled in the art based on this disclosure. The limitations in the claims should be interpreted broadly based on the language used in the claims and not limited to the examples described herein or examples in the prosecution of this application. The examples should be construed as non-exclusive. Furthermore, the steps of the disclosed methods can be modified in any manner, including rearranging steps and / or inserting or deleting steps. Accordingly, it is intended that the specification and examples be considered as exemplary only, with the true scope and spirit being indicated by the appended claims and their full scope of equivalents.

Claims

1. 1. A model-based system for determining a date of a patient event, comprising: accessing a database storing medical records associated with a patient, the medical records including unstructured data; analyzing the unstructured data to identify a plurality of pieces of information within the medical record that are related to a patient event; determining a date associated with each of said plurality of fragments; identifying a plurality of query periods related to the patient's event; and generating, for each of the query periods, a probability of whether an event occurred for the patient during the query period based on the plurality of fragments and the associated dates; At least one processor programmed to A model-aided system comprising:

2. The model-based system of claim 1 , wherein the patient events include at least one of a diagnosis date, a treatment date, or an operation date.

3. The model-aided system of claim 1 , wherein determining the date associated with each of the plurality of fragments comprises identifying a date based on metadata of a document in which the fragment is contained.

4. The model-aided system of claim 1 , wherein determining the date associated with each of the plurality of snippets comprises identifying a date mentioned within the snippet.

5. The model-aided system of claim 1 , wherein the at least one processor is further programmed to generate a vector of a plurality of fragments based on the fragments.

6. 2. The model-assisted system of claim 1, wherein identifying the plurality of query periods comprises identifying a plurality of query dates, the plurality of query periods comprising at least one period for each of the query dates.

7. 7. The model-aided system of claim 6, wherein the at least one time period for each of the query dates includes a time period encompassing the query date, a time period before the query date, and a time period after the query date.

8. The model-based system of claim 6 , wherein the plurality of query dates includes dates spaced one week apart.

9. The model-assisted system of claim 6 , wherein generating the probability comprises evaluating the plurality of fragments over a plurality of time windows for each of the plurality of query dates.

10. 10. The model-aided system of claim 9, wherein evaluating the plurality of fragments across a plurality of time windows comprises processing fragments associated with dates included in the time window for each of the plurality of time windows using one or more aggregation functions.

11. 11. The model-aided system of claim 10, wherein the one or more aggregation functions include at least one of a sum function, a mean function, or a LogSumExp function.

12. The model-aided system of claim 10 , wherein generating the probabilities comprises inputting results of the plurality of functions into a feedforward network.

13. 1. A method for determining a date of an event in a patient, comprising: accessing a database storing medical records associated with a patient, the medical records including unstructured data; analyzing the unstructured data to identify a plurality of pieces of information within the medical record that are related to a patient event; determining a date associated with each of said plurality of fragments; identifying a plurality of query periods related to the patient's event; and generating, for each of the query periods, a probability of whether an event occurred for the patient during the query period based on the plurality of fragments and the associated dates; A method comprising:

14. 14. The method of claim 13, wherein the patient event comprises at least one of a diagnosis date, a treatment date, or an operation date.

15. The method of claim 13 , wherein identifying the plurality of query periods comprises identifying a plurality of query dates, the plurality of query periods comprising at least one period for each of the query dates.

16. 16. The method of claim 15, wherein the at least one period for each of the query dates includes at least one period including the query date, at least one period before the query date, and at least one period after the query date.

17. The method of claim 15 , wherein generating the probability comprises evaluating the plurality of fragments over a plurality of time windows for each of the plurality of query dates.

18. 20. The method of claim 17, wherein evaluating the plurality of fragments across a plurality of time windows comprises processing the fragments associated with dates included in the time window for each of the plurality of time windows using a plurality of functions.

19. 20. The method of claim 18, wherein the plurality of functions includes at least one of a sum function, a mean function, or a LogSumExp function.

20. 1. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform a method for determining a date of a patient event, the method comprising: accessing a database storing medical records associated with a patient, the medical records including unstructured data; analyzing the unstructured data to identify a plurality of pieces of information within the medical record that are related to a patient event; determining a date associated with each of said plurality of fragments; identifying a plurality of query periods related to the patient's event; and generating, for each of the query periods, a probability of whether an event occurred for the patient during the query period based on the plurality of fragments and the associated dates; A non-transitory computer readable medium comprising:

21. 6. The model-aided system of claim 5, wherein generating the vector of the plurality of snippets comprises replacing at least one term in the plurality of snippets with a tokenized representation of the at least one term.

22. The at least one processor further comprises: applying one or more aggregation functions to the plurality of fragment vectors and associated dates to generate a plurality of query outputs for each of the plurality of query periods; generating, for each of the query periods, the probability of whether an event occurred for the patient during the query period based on at least one of the plurality of query outputs; 2. The model-aided system of claim 1, programmed to:

23. The one or more aggregation functions: a first function representing a sum of the plurality of fragment vectors for at least one time window; a second function representing an average of the plurality of fragment vectors for the at least one time window; and a third function for the at least one time window, the third function being different from the first function and the second function; 23. The model-aided system of claim 22, comprising at least one of:

Images