Computer intervention response extraction system

Abstract

Described herein are techniques of automatically extracting intervention responses of a subject from data associated with the subject. The system automatically determines time points (e.g., dates) indicating periods in which intervention responses were determined for subjects, and then uses the time points to identify datasets from which to extract intervention responses. The system extracts intervention responses of the subject from the identified datasets.

Description

COMPUTER INTERVENTION RESPONSE EXTRACTION SYSTEMRELATED APPLICATIONS

[0001] This Application claims priority under 35 U.S.C. 119(e) to U.S. Provision Application Serial No. 63 / 556,341 entitled “COMPUTER INTERVENTION RESPO EXTRACTION SYSTEM,” filed on February, 21, 2024, which application is incorpc herein by reference in its entirety.FIELD

[0002] Described herein are techniques for computer-based extraction of subjects to interventions from datasets associated with the subjects.BACKGROUND

[0003] Clinical record data may be collected about a subject (e.g., a patient) over stored in an electronic health record (EHR). The EHR may be used by a healthcare s) collect and store medical information about multiple different subjects. The clinical n may include various types of information about the subject such as information about history, diagnoses of condition(s), intervention (e.g., medication), intervention respor laboratory data, and / or other types of information.SUMMARY

[0004] Real-world data (RWD) is clinical record data associated with subjects (e. patients). The data may include information about a subject’s health (e.g., diagnosed stage of illness, age, weight, height, allergies, date of birth, and / or other information) delivery of healthcare (e.g., prescribed drugs, medical test history, medical image dat other information). RWD can come from various sources including electronic health (EHRs), insurance claims, monitoring devices (e.g., wearable devices, at home monitRWD can be collected from any number of groups. Insights gained from the data can for use in medical diagnosis, medical treatment, development of treatments (e.g., pro< drugs, and / or other types of treatments), life science research, and / or other purposes.

[0006] An important data point in RWD about a subject is the subject’s response intervention (e.g., chemotherapy) for a condition (e.g., a tumor present in the subject) point is a key measure of intervention efficacy and in regulatory approval of an in ten (e.g., as part of a clinical trial). Described herein are techniques for automatically ext] intervention responses of a subject from data associated with the subject. The system automatically determines time points (e.g., dates indicating periods in which interven responses were determined for subjects) and then uses the time points to identify data which to extract intervention responses. The system extracts the intervention response subject from the identified datasets.

[0007] Some embodiments provide a computer system for automatically extractii intervention responses from data associated with subjects, the computer system comp least one processor; and a plurality of modules executed by the at least one processor, plurality of modules comprising a time point extraction module, a response data idem module, and a response extraction module, wherein: the time point extraction module configured to: access imaging related data associated with a subject; process the ima data using a first trained machine learning (ML) model to obtain a plurality of time p< indicating respective predicted time periods of intervention response determination fc subject), the processing comprising: generate a plurality of sets of features using data image related data; and process the plurality of sets of features using the first trained to obtain the plurality of time points; the response data identification module is config generate, using the data associated with the subject, a dataset collection for each of at of the plurality of time points to obtain a plurality of dataset collections, the generatir comprising: identify, in the data associated with the subject, one or more datasets gen the time point; and include the one or more datasets in the dataset collection; and thefrom the dataset collection; and process the set of features using the second trained M obtain an intervention response of the plurality of intervention responses.

[0008] Some embodiments provide a method for automatically extracting interve responses from data associated with subjects. The method comprises: using at least oi to perform: accessing imaging related data associated with a subject; processing the i related data using a first trained machine learning (ML) model to obtain a plurality of points, the processing comprising: generating a plurality of sets of features using data image related data; and processing the plurality of sets of features using the first train model to obtain the plurality of time points; identifying, from the data associated witl subject, a first collection of one or more datasets generated after a first one of the pirn time points; generating, using the data associated with the subject, a dataset collectioi at least some of the plurality of time points to obtain a plurality of dataset collections, generating comprising: identifying, in the data associated with the subject, one or mo generated after the time point; and including the one or more datasets in the dataset c< and processing the plurality of dataset collections using a second trained ML model t< plurality of intervention responses, the processing comprising, for each of at least son plurality of dataset collections: generating a set of features using data from the datase and processing the set of features using the second trained ML model to obtain an int response of the plurality of intervention responses

[0009] Some embodiments provide a non- transitory computer-readable storage rr storing instructions. The instructions, when executed by at least one processor of a co system, cause the at least one processor to perform a method for automatically extrac intervention responses from data associated with subjects. The method comprises: ac< imaging related data associated with a subject; processing the imaging related data us trained machine learning (ML) model to obtain a plurality of time points, the processcomprising: generating a plurality of sets of features using data from the image relate processing the plurality of sets of features using the first trained ML model to obtaindata associated with the subject, one or more datasets generated after the time point; ; including the one or more datasets in the dataset collection; and processing the plural dataset collections using a second trained ML model to obtain a plurality of intervent responses, the processing comprising, for each of at least some of the plurality of dat; collections: generating a set of features using data from the dataset collection; and pre set of features using the second trained ML model to obtain an intervention response plurality of intervention responses.

[0010] Some embodiments provide a computer system for automatically extractii intervention responses from data associated with subjects. The computer system com least one processor; and at least one non-transitory computer-readable storage mediui instructions that, when executed by the at least one processor, cause the at least one p perform: accessing imaging related data associated with a subject; processing the ima data using a first trained machine learning (ML) model to obtain a plurality of time p< processing comprising: generating a plurality of sets of features using data from the ii related data; and processing the plurality of sets of features using the first trained ML obtain the plurality of time points; identifying, from the data associated with the subj< collection of one or more datasets generated after a first one of the plurality of time p generating, using the data associated with the subject, a dataset collection for each of some of the plurality of time points to obtain a plurality of dataset collections, the ger comprising: identifying, in the data associated with the subject, one or more datasets after the time point; and including the one or more datasets in the dataset collection; i processing the plurality of dataset collections using a second trained ML model to ob plurality of intervention responses, the processing comprising, for each of at least son plurality of dataset collections: generating a set of features using data from the datase and processing the set of features using the second trained ML model to obtain an int response of the plurality of intervention responses.

[0011] Some embodiments provide a computer system for automatically extractiicollection of one or more datasets; and process the first set of features using the parar the at least one model to obtain a first intervention response.

[0012] The foregoing is a non-limiting summary.BRIEF DESCRIPTION OF DRAWINGS

[0013] Various aspects and embodiments will be described with reference to the 1 figures. It should be appreciated that the figures are not necessarily drawn to scale. It< appearing in multiple figures are indicated by the same or a similar reference number figures in which they appear.

[0014] FIG. 1 A illustrates an example intervention response extraction system, a< some embodiments of the technology described herein.

[0015] FIG. IB is a diagram illustrating an example interaction among the modul intervention response extraction system 100 of FIG. 1 A, according to some embodim technology described herein.

[0016] FIG. 1C is a diagram illustrating the extraction of another intervention res according to some embodiments of the technology described herein.

[0017] FIG. ID is a diagram illustrating another example interaction among modi intervention response extraction system 100 of FIG. 1 A, according to some embodim technology described herein.

[0018] FIG. 2A is a diagram illustrating the determination of time points indicatii time periods in which an intervention response was determined for a subject, accordii embodiments of the technology described herein.

[0019] FIG. 2B is another diagram illustrating the determination of time points in predicted time periods in which an intervention response was determined for a subjec to some embodiments of the technology described herein.

[0020] FIG. 3 is a diagram illustrating the extraction of intervention responses foi according to some embodiments of the technology described herein.

[0022] FIG. 4B is a diagram of another machine learning model architecture 410 extraction of intervention responses from data associated with a subject, according to embodiments of the technology described herein.

[0023] FIG. 5A is an example process of extracting intervention responses for a s data associated with the subject, according to some embodiments of the technology d herein.

[0024] FIG. 5B is an example process of using data from a partition storing imag data to determine a time point, according to some embodiments of the technology def herein.

[0025] FIG. 6 is an example process of identifying dataset collections from whicl intervention responses, according to some embodiments of the technology described

[0026] FIG. 7 is an example process of determining an intervention response for ; using data associated with the subject, according to some embodiments of the techno] described herein.

[0027] FIG. 8 is an example process of filtering data extracted from a dataset coll a collection of clinical documents), according to some embodiments of the technolog herein.

[0028] FIG. 9 is a diagram depicting filtering of data for a collection of datasets, ; some embodiments of the technology described herein.

[0029] FIG. 10 is a diagram of an illustrative computer system that may be used i implementing some embodiments of the technology described herein.DETAILED DESCRIPTION

[0030] The inventors have developed a computer intervention response extractioi The intervention response extraction system automatically determines time points (e.i indicating when an intervention response was likely determined for a subject (e.g., bj physician) and uses those time points to identify data that is likely to indicate an interwhen an intervention response was likely determined for a subject. The intervention i extraction system may use the identified date(s) to identify clinical visit note docume in a time period following the identified date(s). The intervention response extraction may process text from the clinical visit note document(s) to automatically extract inte responses of the subject.

[0031] An important data point in real-world data (RWD) about a subject is the si response to an intervention (e.g., chemotherapy, vaccine, drug, surgery, or other inter a condition (e.g., a solid tumor present in the subject, an infection, or other condition' point is a key measure of intervention efficacy and in regulatory approval of an in ten (e.g., as part of a clinical trial). However, conventional computer information extracti are unable to effectively extract intervention responses from clinical data. For examp case of solid tumors, the intervention response of a subject may be expressed using tf Evaluation Criteria in Solid Tumors (RECIST) standard. According to this standard, ; response to intervention to treat a solid tumor may be a complete response (CR), a pa response (PR), stable disease (SD), and progressive disease (PD). Determining a subj response using the RECIST standard involves analysis of radiographic images (e.g., r resonance imaging (MRI) scans, X-Ray scans, computed tomography (CT) scans, an< radiographic images) that cannot be performed by a computer system. Moreover, eve computer system could perform such analysis, radiographic images of a subject are r; accessible. For example, radiographic images are typically not available in electronic record (EHR) data of a subject that may be accessible by an information extraction s) Accordingly, conventional computer information extraction systems are unable to eff extract intervention responses of a subject from available data associated with a subje

[0032] Accordingly, the inventors have developed an intervention response extra* that reliably extracts a subject’s intervention responses from available data associated subject. The inventors recognized that certain datasets in data associated with a subje contain information that indicates a subject’s response to an intervention (e.g., as detetext from the clinical visit note document(s) to output an intervention response indica clinical visit note document(s). Continuing with the example of a tumor, the intervenl response extraction system may process clinical visit note document(s) storing text in physician’s determination of a response to an intervention to treat the tumor.

[0033] Data associated with a subject may include hundreds or thousands of data: clinical visit note documents) and extraction of intervention responses may need to b< for hundreds or thousands of subjects. Given this large amount of data, another chalk extracting intervention responses of subjects is identifying which datasets are likely t< information indicating intervention responses. Conventional information extraction s; unable to identify specific datasets (e.g., specific clinical visit note document(s)) that indicate an intervention response of a subject. As a result, conventional information e systems need to expend a large amount of computing resources to process large amor (e.g., clinical visit note documents) that do not include information about subjects’ in responses. Moreover, conventional information extraction systems are unable to relia the intervention responses of subjects from clinical data because of the inability to idt datasets that are likely to indicate the intervention responses of subjects.

[0034] Accordingly, the inventors have further developed an intervention respons system that can identify datasets, from among data associated with a subject, that are include information indicating an intervention response of the subject. The inventors that datasets (e.g., clinical visit note document(s)) indicating a subject’s intervention j often generated after radiographic imaging is performed on a subject. Accordingly, tf have developed an intervention response extraction system that determines time poi dates) indicating when an intervention response was determined for a subject. The int response extraction system uses imaging related data (e.g., radiology report documen associated with the subject to identify time points indicating when an intervention res likely determined for the subject.

[0035] The intervention response extraction system accesses imaging related dataimaging report documents (e.g., radiology report documents) generated after medical (e.g., radiographic imaging) is performed on a subject. An imaging report document i indicate a time (e.g., a date) when an imaging test was performed, a type of imaging t performed (e.g., MRI scan, X-Ray scan, CT scan, or other type of scan), medical hist subject (e.g., symptom(s), diagnosed condi tion(s)), findings in areas of the subject’s I were scanned, comparison of a finding with a finding from a previous imaging test, a of findings, diagnosis of a condition, recommendation for further testing, and / or othe: information. The intervention response extraction system uses the time point(s) to ide dataset(s) (e.g., clinical visit note document(s)) generated in the time period(s) follow point(s). These dataset(s) are likely to include information indicating intervention res the subject. For example, dataset(s) immediately stored in a database (e.g., an EHR) 1 time when radiographic imaging of a subject’s tumor was performed has a high likeli including data (e.g., text) indicating an intervention response of the subject (e.g., dete physician). The intervention response extraction system uses the identified dataset(s) the intervention responses of the subject.

[0036] In some embodiments, the intervention response extraction system uses a learning model to extract an imaging test date. The imaging test date may indicate a t when an intervention response was determined for a subject and when dataset(s) (e.g. visit note documents) indicating the intervention response were generated (e.g., by ad dataset(s) or updating existing dataset(s)). The machine learning model may be traine data (e.g., text) from imaging related data to output a predicted time point when a rad imaging test was performed on a subject. The intervention response extraction systen the time point to identify dataset(s) (e.g., clinical visit note document(s)) from which intervention response(s). The intervention response extraction system may use anothe learning model to extract an intervention response using the identified dataset(s). The learning model may be trained to process data (e.g., text) from the dataset(s) to outpu predicted intervention response indicated in the dataset(s).machine learning (ML) model (e.g., a first neural network) to obtain a plurality of tin (e.g., dates indicating respective predicted time periods of intervention response detei for the subject). The computer system may be configured to process the imaging relat (1) generating a plurality of sets of features (e.g., numeric vectors) using data from th related data; and (2) process the plurality of sets of features using the first trained ML obtain the plurality of time points. The computer system may be configured to genera the data associated with the subject, a dataset collection (e.g., collection of one or mo visit note documents) for each of at least some of the plurality of time points to obtaii of dataset collections. The computer system may be configured to generate a dataset < for each time point by: (1) identifying, in the data associated with the subject, one or datasets (e.g., clinical visit note documents) generated after the time point; and (2) in< one or more datasets in the dataset collection. The computer system may be configure process the plurality of dataset collections using a second trained ML model to obtair of intervention responses (e.g., each selected from a group consisting of: complete re: partial response, stable disease, progressive disease, and unknown). The computer sy: be configured to process each of at least some of the plurality of dataset collections b generating a set of features using data from the dataset collection; and (2) processing features using the second trained ML model to obtain an intervention response of the intervention responses.

[0038] In some embodiments, the computer system is further configured to gener storing the at least some time points and corresponding intervention responses obtain dataset collections generated using the at least some time points. In some embodimeu generating a dataset storing the at least some time points and corresponding intervent responses obtained using dataset collections generated using the at least some time pc comprises generating a dataset storing :a first time point of the at least some time poii intervention response corresponding to the first time point, the first intervention respc obtained using data from a first one of the plurality of dataset collections, the first datcollections, the second dataset collection comprising one or more datasets generated ; second time point.

[0039] In some embodiments, the computer system may be configured to identify the plurality of time points such that each pair of the subset of time points is separate* a threshold amount of time (e.g., two weeks). In some embodiments, identifying the s time points comprises: identifying one or more of the plurality of time points that are the threshold amount of time after a respective preceding time point of the plurality o points; and filtering out the one or more time points from the plurality of time points the subset of time points.

[0040] In some embodiments, the computer system may be further configured to: imaging related data into a plurality of partitions; wherein generating the plurality of features using the data from the image related data comprises using each of the plural partitions to generate a respective one of the plurality of sets of features. In some eml the imaging related data comprises a plurality of imaging report documents; the plura partitions are associated with a respective plurality of time periods; and dividing the i related data into the plurality of partitions comprises: dividing the plurality of imagin documents into the plurality of partitions by storing, in each of the plurality of part it i < more of the plurality of imaging report documents generated in a respective time peri associated with the partition.

[0041] In some embodiments, identifying, in the data associated with the subject, more datasets generated after the time point comprises: identifying one or more earlie generated datasets (e.g., the three earliest generated datasets) after the time point.

[0042] In some embodiments, the computer system may be further configured to first trained ML model by performing training using training data comprising: sets of generated using imaging related data associated with a plurality of subjects; and time indicating target time point predictions for the sets of features. In some embodiments computer system may be further configured to obtain the second trained ML model b

[0043] In some embodiments, generating the plurality of sets of features using da image related data comprises generating the plurality of sets of features using text ext the image related data. In some embodiments, generating the set of features using dat dataset collection comprises: extracting text from the dataset collection; and generatii features using the text extracted from the dataset collection. In some embodiments, tf extracted from the dataset collection includes a first set of text and the computer systt further configured to: extract a second set of text from one or more datasets generatec or more datasets of the dataset collection; determine a measure of similarity between of text and the second set of text; determine that the measure of similarity meets a thr of similarity; and remove the first set of text from the text extracted from the dataset < when the measure of similarity meets the threshold similarity to obtain a filtered set c wherein generating the set of features using the text extracted from the dataset collect comprises generating the set of features using the filtered set of text.

[0044] The techniques described herein may be implemented in any of numerous the techniques are not limited to any particular manner of implementation. Examples implementation are provided herein solely for illustrative purposes. Furthermore, the disclosed herein may be used individually or in any suitable combination, as aspects < technology described herein are not limited to the use of any particular technique or c of techniques.

[0045] FIG. 1 A illustrates an example intervention response extraction system 10 according to some embodiments of the technology described herein. As shown in the FIG. 1A, the intervention response extraction system 100 accesses data (e.g., clinical associated with a given subject from a datastore 120. The intervention response extra* system 100 processes the data associated with the subject to automatically extract res of the subject to an intervention. For example, the intervention response extraction sy may process the data associated with the subject to automatically extract responses E intervention performed for treatment of a tumor in the subject. As shown in FIG. 1A,

[0046] In some embodiments, the intervention response extraction system 100 m; imaging related data associated with a subject from a datastore storing imaging relate associated with the subject that was generated at different times. In some embodimen intervention response extraction system 100 may access all imaging related data asso< a subject. In some embodiments, the intervention response extraction system 100 ma; imaging related data associated with a subject that meets one or more criteria. For ex; intervention response extraction system 100 may filter imaging related data associate subject based on a category of a document, a title of a document, and / or a timestamp document. To illustrate, the intervention response extraction system 100 may access i related data with the following attributes: (1) a particular document category (e.g., “R Nuclear medicine”); (2) a title containing certain strings (e.g., “xr”, “ray”, “x-r”, “x-r; and / or “venous”); (3) a timestamp greater than or equal to a first date (e.g., a line of ti date) plus a number of days (e.g., 30 days) and less than or equal to a second date (eq therapy end date).

[0047] In some embodiments, the intervention response extraction system 100 m; imaging related data that was generated during a particular time period. For example, intervention response extraction system 100 may access imaging related data that wai after the start date of a line of therapy (LOT) (e.g., chemotherapy, radiation, surgery, prescription, and / or a combination thereof) and before the end date of the LOT. The c partitioning module 102 may determine the start and end dates of a LOT from fields < data records associated with the subject (e.g., by querying for values of the start and e the LOT). As another example, the intervention response extraction system 100 may imaging related data that was generated between the diagnosis date of a condition am date. In some embodiments, the intervention response extraction system 100 may acc related data that was generated in a user-specified time period. For example, the inter response extraction system 100 may receive user input (e.g., through a graphical user (GUI)) indicating a time period (e.g., a date range) and access imaging related data gfa time period of 30-60 days, 60-90 days, 90-120 days, 120-150 days, 150-200 days, 2 days, 250-300 days, 300-360 days, or another suitable time period.

[0048] In some embodiments, the data partitioning module 102 may divide the ac imaging related data into multiple partitions. A data partition may also be referred to “chunk”. The data partitioning module 102 may divide the imaging related data assoc the subject into partitions by organizing the data into the partitions based on times wl imaging related data was generated (e.g., as indicated by timestamps associated with related data). Each of the data partitions may include imaging related data (e.g., radio documents) generated in a respective time period associated with the data partition, h embodiments, each partition may include imaging related data generated in a respecti period of at least 1-2 weeks, 2-4 weeks, 4-6 weeks, 6-8 weeks, 8-10 weeks, or anothe amount of time. For example, each of the data partitions may include imaging related generated in a time period of at least two weeks. In some embodiments, the data in a < partition may include data in addition to imaging related data, where the data is genei time period associated with the data partition. For example, the data in the data partiti include clinical visit note documents generated in the time period associated with the partition (e.g., as indicated by timestamps of the clinical visit note documents).

[0049] In some embodiments, the imaging related data may include radiology ref documents. The data partitioning module 102 may divide the radiology report docum multiple partitions by: (1) sorting the radiology report documents based on time of ge (e.g., indicated by timestamps associated with the documents); and (2) dividing the rr report documents into multiple data partitions associated with respective time periods partition may include radiology report documents generated in a time period associate data partition. In some embodiments, the time periods may be a sequence of consecut periods. In some embodiments, each of the time periods may be at least a threshold a; time (e.g., 1 week, 2 weeks, 3 weeks, 4 weeks, 5 weeks, 6 weeks, or another suitable time). As an illustrative example, the data partitioning module 102 may generate a firpartition associated with a second time period by: (1) identifying a second radiology j document d2 that is the earliest radiology report document generated after the thresho of time from generation of the first radiology report document di; and (2) including, i second data partition, the second radiology report document d2 and all subsequent rad report documents generated within the threshold amount of time from generation of t radiology report document d2. Thus, each data partition may include radiology report generated in an associated time period that has a length of at least the threshold amou

[0050] In some embodiments, the data partitioning module 102 may allocate ima; data to partition based on a time associated with the imaging related data. For exampl partitioning module 102 may use timestamps included in datasets (e.g., radiology rep documents) of the imaging related data to allocate the datasets into specific partitions illustrate, the data partitioning module 102 may access a timestamp of a stored radioli document, identify a time period containing the time indicated by the timestamp, and radiology report document to a partition associated with the time period.

[0051] In some embodiments, the data partitioning module 102 may store imagin data associated with the subject organized based on the partitions. For example, the d partitioning module 102 may store each partition as a row of a database table. To i 11 LI row of the table may store imaging related data with a timestamp in a time period def start date and an end date. For example, a first row may store imaging related data wi timestamp indicating a date between January 1, 2023, and January 14, 2023. A secon store imaging related data with a timestamp indicating a date between January 15, 20 January 30, 2023.

[0052] As indicated by the dashed lines around the data partitioning module 102 < in some embodiments, the intervention response extraction system 100 may not inclu partitioning module 102. In such embodiments, imaging related data associated with may not be partitioned. The time point extraction module 104 may be configured to u accessed imaging related data associated with a subject unpartitioned.response was determined for the subject. In some embodiments, the time point extrac 104 may process imaging related data associated with a subject to obtain time points predicted time periods of intervention response determination for the subject. The tin extraction module 104 may process the imaging related data by: (1) generating sets o using data (e.g., text) from the imaging related data; and (2) processing the sets of fee a trained machine learning model to obtain the time points indicating the predicted tii of intervention response determination. In some embodiments, each time point may ii predicted time period subsequent to the time point in which an intervention response determined for the subject.

[0054] In some embodiments, the imaging related data associated with a subject i multiple datasets and the time point extraction module 104 may determine a time poii corresponding to each dataset. For example, each of the datasets may be an imaging r document (e.g., a radiology report document). The time point extraction module 104 generate a set of features for each imaging report document (e.g., by extracting text fi imaging report document and generating the set of features using the extracted text). ' point extraction module 104 may process the set of features generated for each imagi: document using a trained machine learning model to obtain a respective time point, h embodiments, the time point extraction module 104 may determine a time point for e in imaging related data associated with a subject. In some embodiments, the time poii module 104 may determine a time point for each dataset in a subset of imaging relate associated with a subject. For example, the time point extraction module 104 may det time point for each dataset generated in a particular time period (e.g., between an LO' end date, a user- specified time period, or other time period).

[0055] In some embodiments, the time point extraction module 104 may use data determined by the data partitioning module 102 to determine time points (e.g., dates) predicted time periods when intervention responses were determined for the subject. ’ point extraction module 104 may use data within a data partition to determine a timedocument(s)). The time point extraction module 104 may extract one or more snippet from the unstructured data and generate a set of features using the snippet(s) of text e from the unstructured data. As another example, the data partition may include struct (e.g., values of fields in a database). The time point extraction module 104 may use si data to generate the set of features. As another example, the time point extraction moi may use a combination of unstructured data (e.g., text snippets) and structured data (e database field values) to generate the set of features.

[0056] In some embodiments, the machine learning model used by the time point module 104 may be trained to output a time point. For example, the machine learning be trained to output a time point indicative of a predicted time period in which an inte response was determined for the subject. In some embodiments, the machine learning output a classification indicating the time point. For example, the machine learning rr output scores associated with multiple different time points (e.g., dates). The time poi extraction module 104 may output a time point with the greatest associated score as a of an intervention response determination. In some embodiments, the machine learnii may be a neural network model. The parameters of the neural network may be trainee applying a supervised learning technique (e.g., stochastic gradient descent) to a set of data. For example, the training data may include sets of features and corresponding tc point labels. The neural network may be trained by applying stochastic gradient desce training data.

[0057] In some embodiments, the time point extraction module 104 may filter tin obtained using a trained machine learning model to obtain a subset of time points. In embodiments, the time point extraction module 104 may obtain the subset of time po that each pair of time points in the subset of time points is separated by at least a th re: amount of time. For example, each pair of time points in the subset of time points ma separated by at least 1 week, 8 days, 9 days, 10 days, 11 days, 12 days, 13 days, 2 we days, 16 days, 17 days, 18 days, 19 days, 20 days, 3 weeks, 22 days, 23 days, 24 daystime after a respective preceding time point in a set of time points obtained using the machine learning model; and (2) filtering out the identified time point(s) to obtain the time points. The time point extraction module 104 may provide the subset of time po: filtered set of time points) to the response data identification module 106 for identific intervention response data.

[0058] In some embodiments, the response data identification module 106 may u points determined by the time point extraction module 104 to identify intervention re: to use in determining an intervention response. In some embodiments, the response d identification module 106 may use a given time point to generate a corresponding dat collection of one or more datasets to use in the extraction of an intervention response response data identification module 106 may identify, in data associated with a subje< clinical record data), dataset(s) generated after the time point, and include the identifi dataset(s) in the dataset collection. For example, the response data identification mod may use the time point to identify one or more clinical visit note documents to use in an intervention response. In some embodiments, the response data identification mod may identify a particular number of the earliest generated datasets generated after the and include the identified datasets in a dataset collection corresponding to the time pc some embodiments, the response data identification module 106 may exclude dataset before or after the identified datasets from the dataset collection. For example, the res identification module 106 may identify a particular number of clinical visit note docu oncology clinical visit (OCV) note documents) generated after the time point (e.g., as by timestamps of the clinical visit note documents). In some embodiments, the respoi identification module 106 may identify the first 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or other su number of the earliest datasets generated after the time point. For example, the respoi identification module 106 may identify the 3 earliest clinical visit note documents gei a time point.

[0059] In some embodiments, the response data identification module 106 may stdocuments generated after a respective time point. The rows of the database table ma determine intervention responses for each time point.

[0060] In some embodiments, the response extraction module 108 may use interv response data identified by the response extraction module 108 to determine the inter responses 130. The response extraction module 108 may use a dataset collection iden a time point to determine an extracted intervention response using data from the data1collection. In some embodiments, the response extraction module 108 may generate i features (e.g., tokens) using data from the dataset collection and process the set of fea a trained machine learning model to output a corresponding extracted intervention res example, the response extraction module 108 may extract snippets of text from a set < visit note document(s) in a dataset collection and generate a set of features using the f text (e.g., by generating a set of tokens). The response extraction module 108 may pr< of features using the trained machine learning model to output an intervention respon

[0061] In some embodiments, the response extraction module 108 may use one o previous intervention responses as input features. For example, in addition to tokens j from text snippets extracted from clinical visit note document(s), the response extract 108 may include one or more values in a set of features that indicate previous interve response(s) determined by the response extraction module 108 (e.g., for one or more intervention response corresponding to time points that precede the time point for wh intervention response is being determined). The previous intervention response(s) ma metadata included as features for determining an intervention response. In some embi the response extraction module 108 may filter out data (e.g., text snippets) that was ir dataset(s) generated prior to dataset(s) in a dataset collection corresponding to a time data that was also included in previously generated dataset(s) may degrade prediction intervention response by the machine learning model (e.g., because the data may be u an intervention response determined in the time period after the time point).

[0062] In some embodiments, an intervention response determined by the respon:highest score to be the extracted intervention response for a time point. To illustrate, i types for an intervention response for the treatment of a tumor may be a complete res partial response (PR), stable disease (SD), progressive disease (PD), or unknown. In f embodiments, the response extraction module 108 may be configured to use different response categories for different types of conditions (e.g., different types of tumors, d pathogens, and / or other types of conditions).

[0063] In some embodiments, the response extraction module 108 may generate ; dataset including a set of time points (e.g., determined by the time point extraction m and intervention responses corresponding to the time points (e.g., determined using d collections corresponding to the time points). For example, the output dataset may be storing time points and corresponding intervention responses. The table may include: row storing a first time point and a corresponding first intervention response that was using data from a first collection of dataset(s) generated after the first time point; and second row storing a second time point subsequent to the first time point and a corres second intervention response that was obtained using data from a second collection o generated after the second time point. In some embodiments, the response extraction may filter the output dataset to obtain a filtered output dataset. For example, the respc extraction module 108 may filter the output dataset to include time points and corresj intervention responses in a particular time period (e.g., between an LOT start and end user-specified time period, or other time period).

[0064] In some embodiments, the datastore 110 of the intervention response extn system 100 may store model parameters. The model parameters may include paramet machine learning model used by the time point extraction module 104 in predicting ti indicative of time periods in which an intervention response was determined for a sut model parameters may include parameters of a machine learning model used by the r extraction module 108 in predicting intervention responses. The datastore 110 stores data. The extraction data may include data partitions, time points, dataset collectionsoutput dataset(s) generated by the response extraction module 108 (e.g., storing time corresponding intervention responses for a subject).

[0065] In some embodiments, the datastore 110 may comprise data storage hardv the model parameters and extraction data. For example, the datastore 110 may compr more hard drives storing the data associated with the subjects. In some embodiments, datastore 110 may be a distributed database storing the data.

[0066] In some embodiments, the intervention response extraction system 100 m; data associated with a subject from the datastore 120. The datastore 120 may store da associated with multiple different subjects including the subject for whom interventic extraction is being performed. For example, the datastore 120 may store clinical recoi associated with different subjects (e.g., accessed from an electronic health record (EF database). Data associated with a subject that is stored in the datastore 120 may incln related data, textual notes, medical history data, immunizations, laboratory data, med data, and / or other data related to the subject’s health. For example, the data associate< subject may include radiology report documents generated after the performance of r; imaging on the subject (e.g., MRI, X-Ray, CT, ultrasound, and / or other radiographic As another example, the data associated with the subject may include clinical visit no documents. A clinical note may, for example, include text describing a diagnosis or ii response determined by a clinician (e.g., a physician). To illustrate, a clinical note ms text describing a response of a subject to intervention for the treatment of a tumor as i by a clinician.

[0067] In some embodiments, the datastore 120 may comprise data storage hardv the data associated with the subjects. For example, the datastore 120 may comprise oi hard drives storing the data associated with the subjects. In some embodiments, the d may be a database storing the data associated with the subjects. The database may be database storing the data associated with the subjects. Although in the example embo FIG. 1A the datastore 120 is shown separate from the intervention response extractioiusing the computer system 1000 described herein with reference to FIG. 10. In some embodiments, the modules 102, 104, 106, 108 may be implemented as respective setf executable instructions stored in memory of the computer system. In some embodime datastore 110 may be memory of the computer system.

[0069] FIG. IB is a diagram illustrating an example interaction among the modul 106, 108 of the intervention response extraction system 100 of FIG. 1A, according to embodiments of the technology described herein. The example of FIG. IB shows hoy intervention response 130 A of the intervention responses 130 may be extracted for a

[0070] In the example of FIG. IB, the intervention response extraction system ac of imaging related data 120A (e.g., radiology report documents) associated with a sul from datastore 120). In some embodiments, the imaging related data 120A may inclu imaging related data associated with a subject. In some embodiments, the imaging re] 120 A may be a subset of imaging related data associated with the subject. The intervt response extraction system 100 may access a subset of imaging related data stored du particular time period. In some embodiments, the particular time period may be speci input (e.g., through a GUI). In some embodiments, the particular time period may be automatically determined by the intervention response extraction system 100. For ext intervention response extraction system 100 may: (1) determine a time point (e.g., a c the subject was diagnosed with a condition (e.g., a tumor); and (2) select a subset of i related data associated with the subject generated after the time point.

[0071] The data partitioning module 102 partitions the imaging related data 1204 illustrated in FIG. IB, the data partitioning module 102 determines multiple data part each associated with a respective one of time periods 124A, 124B, 124C, 124D. In sc embodiments, the periods 124A, 124B, 124C, 124D may be time periods of at least a amount of time (e.g., of at least two weeks). Each partition may include imaging relat generated during its associated time period (e.g., as indicated by timestamps associate data). In some embodiments, the data partitioning module 102 may include, in each d

[0072] The time point extraction module 104 accesses data from a first partition z with time period 124A. For example, the time point extraction module 104 may acce* related data in the first partition. As another example, the time point extraction mode access clinical visit note documents in the first partition in addition to the imaging rel The time point extraction module 104 uses the partition data 122 A to determine a tim 126 A of an intervention response determination. The time point extraction module If the partition data 122A to: (1) generate a set of features (e.g., by extracting text snipp partition data 122A and generating a set of tokens therefrom as the set of features); ai provide the set of features as input to a trained machine learning model to obtain out]- indicating the time point 126A. In some embodiments, the time point 126A may indi< predicted time period in which an intervention response was determined for the subje

[0073] As shown in FIG. IB, the response data identification module 106 accesse point 126A. The response data identification module 106 may use the time point 126. identify a dataset collection 124A from among datasets 124 associated with the subje extracting an intervention response (e.g., to include in a dataset collection). For exam identified dataset collection 124 A may be those datasets that the response data identil module 106 has determined have a high likelihood of including data indicating an int response of the subject (e.g., determined by a clinician). In some embodiments, the d; may include clinical visit note documents associated with the subject. The response d identification module 106 may use the time point 126 A to identify a subset of the clir note documents to use in extracting an intervention response. In some embodiments, response data identification module 106 may identify a certain number of earliest clir note documents generated after the time point 126 A and include the identified clinica documents in the dataset collection 124 A. For example, the response data identificatr 106 may identify the three earliest clinical visit note documents generated after the til 126A (e.g., as indicated by time points associated with the clinical visit note documei

[0074] The response extraction module 108 may use the dataset collection 124 Aproviding the set of features as input to a trained machine learning model to obtain or indicating the extracted response 130A (e.g., a classification into one of multiple resp categories). The response extraction module 108 may store the extracted response 131an output dataset and / or the datastore 110).

[0075] FIG. 1C is a diagram illustrating extraction of another intervention respon according to some embodiments of the technology described herein. As shown in Fit time point extraction module 104 access data 122B from a second partition associate< period 124B and uses the data 122B to determine a time point 126B indicating a pred period in which an intervention response was determined for the subject. The respons identification module 106 accesses the time point 126B and uses it to generate datase 124B from among the datasets 124 associated with the subject (e.g., by identifying th earliest clinical visit note documents generated after the time point 126B and includir the dataset collection 124B). The response extraction module 108 uses the datasets in collection 124B to determine the extracted response 130B.

[0076] It should be appreciated that although the response extractions illustrated i and 1C are illustrated as occurring sequentially, in some embodiments, the responses 130B may be extracted in parallel. Partition data 122A and partition data 122B may t in parallel to obtain time points 126 A, 126B. The time points 126A, 126B may be use response data identification module 106 in parallel to identify dataset collections 124 The response extraction module 108 may determine the extracted responses 130A, 12 parallel using the identified dataset collections 124A, 124B.

[0077] FIG. ID is a diagram illustrating another example interaction among modi intervention response extraction system 100 of FIG. 1 A, according to some embodim technology described herein. In the example of FIG. ID, the time point extraction me uses imaging related data 120 A accessed by the intervention response extraction systi determine time points indicating time periods of intervention response determination subject. The time points include time points 140A, 140B, 140C, 140D, 140E, 140F. Iextracting text from the dataset and processing the text using a trained machine learni obtain the corresponding time point).

[0078] As illustrated in the example of FIG. ID, the time point extraction module filters the time points to generate a subset of time points that are used by the response identification module 106. Example techniques of filtering the time points are descrit with reference to FIG. 1 A. For example, the time point extraction module 104 may fi points 140A, 1406, 140C, MOD, 140E, 140F such that each pair of time points in the time points is separated by at least two weeks. In the example of FIG. ID, the time pc extraction module 104 filters out time points 1406, 140D, 140F (e.g., because they ai two weeks of respective preceding time points 140A, 140C, 140E). The filtered set o1 140 A, 140C, 1406 may be provided to the response data identification module 106.

[0079] As illustrated in the example of FIG. ID, the response data identification i uses the subset of time points 140A, 140C, 140E to identify respective dataset collect 142C, 142E. For example, the response identification module 106 may identify the tl generated datasets after each of the time points 142A, 142C, 142E and include the ide datasets in respective dataset collections 142A, 1426, 142C. The response extraction uses the identified dataset collections 142A, 1426, 142C to determine respected ext re intervention responses 144 A, 1446, 144C (e.g., by generating a set of features using i each dataset collection and processing the set of features using a trained machine lear to obtain a corresponding intervention response). The response extraction module 10! dataset collection 142A to obtain extracted response 144 A, dataset collection 1426 tc extracted response 1446, and dataset collection 142C to obtain extracted response 14

[0080] It should be appreciated that although the response extraction illustrated ir are illustrated as occurring in parallel, in some embodiments, the responses 144A, 14 may be extracted sequentially. Imaging related data may be processed sequentially to points 140A, 1406, 140C, 1400, 140E, 140F. The time points 140A, 140C, HOE ma by the response data identification module 106 sequentially to identify dataset collectof the technology described herein. The determination illustrated in FIG. 2A may be ] by the intervention extraction system 100 of FIG. 1A. For example, the determinatior performed by the intervention extraction system 100 using the data partitioning mode time point extraction module 104.

[0082] As shown in FIG. 2A, the system access data 200 associated with a subjec clinical record data from an EHR). The data 200 may include imaging related data (e. radiology report documents) associated with the subject. The system may identify a p 200 A of the imaging related data to use in determining the time points 206A, 206B, 2 some embodiments, the system may identify imaging related data generated in a parti window (e.g., a certain number of days in the past, a number of days after diagnosis c condition, or a specific user- specified date range). For example, the system may idem radiology report documents with a timestamp within the particular time window as th related data 200A.

[0083] After identifying the imaging related data 200A, the system partitions the into multiple partitions 202A, 202B, 202C. The system may generate the partitions 2( 202C such that each partition represents a respective time period. In some embodime periods may be of equal length (e.g., two weeks). Each of the partitions 202A, 202B, include imaging related data generated in the time period that the partition represents example, the partition 202A may include radiology report documents with a timestan two-week period, the partition 202B may include radiology report documents with a 1 in a second two-week period subsequent to the first two-week period, and the partitio include radiology report documents with a timestamp in a third two-week period subf the second two-week-period. In some embodiments, the system may further include, the partitions 202A, 202B, 202C, data in addition to the imaging related data of the p example, the system may include documents (e.g., clinical visit note documents) gene time period represented by the partition.

[0084] The system may use the data of each of the partitions 202A, 202B, 202C ttime point prediction model 204. For example, the system may extract text snippets fi more radiology report documents by identifying keyword(s) in the radiology report d and extracting segments of text (e.g., a predefined number of words or characters) fol and / or preceding the keyword(s). Example keywords that may be identified in imagir data (e.g., radiology report documents) include “imaging”, “images”, “scan”, “pet”, “ “evidence”, “response”, “cr”, “remission”, “stable”, “larger”, “bigger”, “new”, “wors “progress”, and / or other keywords. The system may tokenize the extracted text snipp a token vector. The system may further determine a numeric representation of the tok For example, the system may determine a word embedding for each token that is a ve number values (e.g., a vector of 128 real numbers) representing the token. The numer representation of the token vector (e.g., a numeric vector) may form a set of features i system may input to the time point prediction model 204.

[0085] In some embodiments, the time point prediction model 204 may be a mac] learning model trained to predict a time point (e.g., a date) for an input set of features predicted time point may indicate a predicted time period in which an intervention re: determined for the subject. For example, the input set of features may be a real value of a vector (e.g., a token vector) generated using text snippets extracted from a data p The machine learning model may be trained using training data. The training data ma sets of input features and labels indicating target time points for the sets of input feati example, the training data may include: (1) sets of features generated from sets of rad report documents generated in a time period (e.g., a period of at least two weeks); am corresponding date labels that are to be output by the machine learning model for eac features. A date label corresponding to a set of radiology report documents may be a predetermined date of an intervention response determination (e.g., performed by a p] after generation of the set of radiology report documents. For example, the date label: dates manually extracted by analyzing sets of radiology report documents. In some embodiments, each set of features in the training data may also be generated using ch

[0086] In some embodiments, the machine learning model may be trained by app supervised learning technique (e.g., stochastic gradient descent) to a set of training dr machine learning model may be trained by: (1) providing the sets of features as input machine learning model to obtain time point predictions; (2) comparing the time poir target time points; and (3) updating parameters of the machine learning model based < difference between the time points and the target time points. In some embodiments, supervised learning technique may be an iterative technique performed to minimize a function. For example, stochastic gradient descent may be applied to minimize a loss based on a difference between the time points and the target time points.

[0087] In some embodiments, the time point prediction model 204 may include a network , a logistic regression model, a linear regression model, a regression model, ; forest model, a K-Nearest Neighbor (KNN) model, a K-Means model, a decision tree cox proportional hazards regression model, a Naive Bayes model, a support vector m (SVM) model, and / or other suitable machine learning model. For example, the time ]' prediction model 204 may be a neural network model trained to predict a time point I set of features. The neural network may include one or more recurrent neural networl layers, one or more long short-term memory network (LSTM) layers, one or more co neural network (CNN) layers, one or more feedforward neural network layers, and / or more other suitable neural network layers of another type. The system may provide tl features (e.g., a vector or matrix of values) as input to the neural network to obtain a corresponding output indicating a time point. In some embodiments, the output may i classification result. For example, the output may include probabilities associated wit different possible time points (e.g., dates in a year). The system may determine the til be the one associated with the greatest probability. For example, the time point predic 204 may be trained to receive, as input, a tokenized snippet vector and output probab an intervention response was determined at different time points (e.g., dates). Examp] learning models that may be used for the time point prediction model 204 are describpartition 202A, time point 206B using data from partition 202B, and time point 206C from partition 202C. The time points 206A, 206B, 206C may subsequently be used ir intervention responses of the subject (e.g., as described herein with reference to FIG.

[0089] As an illustrative example, the identified imaging related data 200A assoc the subject may be radiology report documents generated after radiographic imaging on a subject for a tumor. Each of the partitions 202A, 202B, 202C may include radio! documents generated during the time period that the partition represents. In some eml the partitions 202A, 202B, 202C may each include clinical visit note documents gene during the time period represented by the time period. For each of the partitions 202 202C, the system may generate a corresponding set of features and provide the set of input to the time point prediction model 204 to obtain time points 206A, 206B, 206C points 206 A, 206B, 206C may indicate predicted time periods in which an interventk was determined for the subject (e.g., by a physician).

[0090] FIG. 2B is another diagram illustrating the determination of time points 2 216C, 216D, 216E indicating predicted time periods in which an intervention respons determined for a subject, according to some embodiments of the technology describes The determination illustrated in FIG. 2B may be performed by the intervention extrac 100 of FIG. 1A. For example, the determination may be performed by the interventio system 100 using the time point extraction module 104. In contrast to the example en of FIG. 2 A, in the example embodiment of FIG. 2B, imaging related data 200A is noi prior to determination of time points using the imaging related data 200A.

[0091] The imaging related data 200A may be identified from the data associated subject 200 as described herein with reference to FIG. 2A. As shown in FIG. 2B, the includes datasets 212 (e.g., imaging report documents) that are each processed using point prediction model 204. The system may use data (e.g., extracted text) from each datasets 212 to generate a set of features to provide as input to the time point predict! 204. For example, the system may extract text snippets from one of the datasets 2121other keywords. The system may tokenize the extracted text snippets to obtain a tokei The system may further determine a numeric representation of the token vector. For e system may determine a word embedding for each token that is a vector of real numb (e.g., a vector of 128 real numbers) representing the token. The numeric representatic token vector (e.g., a numeric vector) may form a set of features that the system may i time point prediction model 204.

[0092] As shown in the example embodiment of FIG. 2B, the system obtains tim< 216A, 216B, 216C, 216D, 216E using the time point prediction model 204. In some embodiments, each of the time points 216A, 216B, 216C, 216D, 216E may be obtain respective set of features generated using data from a respective one of the datasets 2 Although not illustrated in FIG. 2B, in some embodiments, the system may filter the 216A, 216B, 216C, 216D, 216E to obtain a subset of time points (e.g., as illustrated i example of FIG. ID). The subset of time points may be used to identify data to use in intervention responses for a subject.

[0093] FIG. 3 is a diagram illustrating the extraction of intervention responses 30 304C for a subject, according to some embodiments of the technology described here extraction illustrated in FIG. 3 may be performed by the intervention extraction syste FIGs. 1A-1B. For example, the prediction may be performed by the intervention extr; system 100 using response data identification module 106 and response extraction m<

[0094] In the example of FIG. 3, the system uses the time points 306A, 306B, 30< identify datasets from which to extract intervention responses of a subject. As shown the system has access to datasets 300 associated with the subject. For example, the ds may be clinical visit note documents associated with the subject. As another example datasets 300 may be clinical visit note documents with a particular attribute (e.g., cliu note documents with a particular category tag). The system uses the time points 3064 306C to identify respective subsets of the datasets 300 associated with the subject. In embodiments, the system may identify a particular number of the earliest generated dgenerates a dataset collection corresponding to each of the time points 306A, 306B, 3 dataset collection may include the datasets identified based on a respective time poinl three earliest datasets generated after the time point). In the example of FIG. 3, the sy generated dataset collection 300A based on time point 306A, dataset collection 300B time point 306B, and dataset collection 300C based on time point 206C. For example the dataset collections 300A, 300B, 300C may be a set of clinical visit note document based on respective time points 306A, 306B, 306C.

[0095] The system may use the data of each of dataset collections 300A, 300B, 31extract a corresponding one of the intervention responses 304A, 304B, 304C. The sys use a response prediction model 302 to determine the intervention responses 304A, 31As indicated by the arrows between the dataset collections 300 A, 300B, 300C and th< prediction model 302, the system may use data from each of the datasets to generate ; features to provide as input to the response prediction model. For example, the systen extract text snippets from one or more clinical visit note documents by identifying ke the clinical visit note document(s) and extracting segments of text (e.g., a predefined words or characters) following and / or preceding the keyword(s). Example keywords identified include “response”, “remission”, “stable”, “larger”, “bigger”, “new”, “won “progress”, and / or other keywords. The system may tokenize the extracted text snipp a token vector. The system may further determine a numeric representation of the tok For example, the system may determine a word embedding for each token that is a ve number values (e.g., a vector of 128 real numbers) representing the token. The numer representation of the token vector (e.g., a numeric vector) may form a set of features i system may input to the response prediction model 302.

[0096] The system may apply filtering to text snippets extracted from a dataset c< prior to generating a set of features. In some embodiments, the system may filter out snippets that appeared in a previously processed dataset collection. For example, clin note documents may include text copied from previous clinical visit note documents iare unlikely to include information about a current intervention response determi natk such text snippets may degrade the accuracy of intervention response predictions mat response prediction model 302. In some embodiments, the system may determine a r similarity between text snippets extracted from a dataset collection and text snippets t from previously processed dataset collection(s). For example, the system may determ measure of similarity to be a percentage of overlap between text snippets. The systerr remove a text snippet with a threshold measure of similarity (e.g., a threshold percent overlap) with a text snippet extracted from a previously processed dataset collection, may remove a text snippet with a measure of similarity relative to a text snippet extra previously processed dataset collection of at least 50-60%, 60-70%, 70-80%, 80-90% or other suitable similarity threshold. For example, the system may remove text snip]- measure of similarity of at least 80% relative to a text snippet extracted from a previc processed dataset collection.

[0097] In some embodiments, the response prediction model 302 may be a machi model trained to output a predicted intervention response for an input set of features, example, the input set of features may be a real value embedding of a vector (e.g., a t vector) generated using text snippets extracted from a dataset collection. The machint model may be trained using training data. The training data may include sets of input and labels indicating target intervention responses for the sets of input features. For e training data may include: (1) sets of features generated from a dataset collection (e.g collection of clinical visit note documents); and (2) corresponding intervention respoi for each set of features. An intervention response label corresponding to a collection < visit note documents may be a predetermined intervention response. For example, the intervention response may be manually extracted by analyzing the clinical visit note < In some embodiments, each set of features in the training data may include an indicat previously determined intervention response. For example, a set of features may be g using clinical visit note documents generated after a certain date, and may include anmachine learning model to obtain predicted intervention responses; (2) comparing the intervention responses to the label intervention responses; and (3) updating parametei machine learning model based on a difference between the predicted intervention res] the target intervention responses. In some embodiments, the supervised learning tech: be an iterative technique performed to minimize a loss function. For example, stocha; descent may be applied to minimize a loss function based on a difference between the intervention responses and the label intervention responses.

[0099] In some embodiments, the response prediction model 302 may include a u network, a logistic regression model, a linear regression model, a regression model, a forest model, a K-Nearest Neighbor (KNN) model, a K-Means model, a decision tree cox proportional hazards regression model, a Naive Bayes model, a support vector m (SVM) model, and / or other suitable machine learning model. For example, the respoi prediction model 302 may be a neural network model trained to predict a time point I set of features. For example, the neural network may include a recurrent neural netwc a long short-term memory network (LSTM), a convolutional neural network (CNN), feedforward neural network, and / or another type of neural network. The system may set of features (e.g., a vector or matrix of values) as input to the neural network to obi corresponding output indicating a predicted intervention response. In some embodim output may indicate a classification result. For example, the output may include prob; associated with various different intervention response categories. The system may d< predicted intervention response to be the response category associated with the greate probability. Example machine learning models that may be used for the response pre< model 302 are described in U.S. Patent Application Publication No. US 2021 / 002789 on January 28, 2021, which is incorporated by reference herein. Descriptions of exan machine learning models are provided in the patent application publication with refer FIGs. 4A, 4B and 5 of the patent application publication and are incorporated herein 1 reference. Example techniques of using a machine learning model are described in pafrom dataset collection 300B, and intervention response 304C using data from datase 300C.

[0101] As an illustrative example, the datasets 300 associated with the subject r oncological clinical visit note documents with text authored by clinician(s) (e.g., phy; Each of dataset collections 300A, 300B, 300C may be a set of the three earliest oncol clinical visit note documents generated after respective time points 306A, 306B, 306( collection of oncological clinical visit note documents, the system generates a set of 1 using text from the oncological clinical visit note documents. The system provides th features as input to the response prediction model 302 to obtain a predicted interventi response.

[0102] FIG. 4A is a diagram of a machine learning model architecture 400 for exl intervention responses from data associated with a subject, according to some embod the technology described herein. In some embodiments, the machine learning model ; 400 of FIG. 4 A may be used by the intervention response extraction system 100 desc with reference to FIGs. 1A-1D.

[0103] As shown in FIG. 4A, the machine learning model architecture 400 includ point prediction model 204 described herein with reference to FIGs. 2A-2B and the n prediction model 302 described herein with reference to FIG. 3. The time point predii 204 includes parameters (e.g., neural network weights) trained to process sets of featr generated from data partitions (e.g., partitions 202 A, 202B, 202C) to output time poii time points 206A, 206B, 206C). The response prediction model 302 includes paramei to process sets of features generated from dataset collections (e.g., dataset collections 300B, 300C) identified based on time points output by the time point prediction modi output predicted intervention responses (e.g., intervention responses 304A, 304B, 30

[0104] In some embodiments, each of the time point prediction model 204 and th prediction model 302 may be trained individually (e.g., as described herein with refer FIG. 2A and FIG. 3). The trained models 204, 302 may then be combined to form theincluding imaging related data (e.g., radiology report documents) and, optionally, oth (e.g., clinical visit note documents) generated in a respective time period; (2) a time f corresponding to the first set of features indicating a target time point to be predicted point prediction model 204; (3) a second set of features generated from a collection o (e.g., clinical visit note documents) generated after the time point label; and (4) an int response label corresponding to the second set of features indicating a target interven response to be predicted by the response prediction model 302. The parameters of the 204, 302 may be trained by applying a supervised learning technique (e.g., stochastic descent) to the training data. The training data samples may be used to determine pre< outputs of the models 204, 302 for input sets of features, compare the predicted outpr models 204, 302 to their corresponding labels, and update parameters of the models 2 based on a difference between the predicted outputs and the labels.

[0105] The trained parameters of the time point prediction model 204 and the par the response prediction model 302 may be stored in memory for use in intervention n extraction. For example, the parameters of the models 204, 302 may be stored in data the intervention extraction response system 100 described herein with reference to FI The trained parameters may be used to process data (e.g., clinical record data comprif radiology report documents and clinical visit note documents) associated with a subje intervention responses of the subject.

[0106] FIG. 4B is a diagram of another machine learning model architecture 410 extraction of intervention responses from data associated with a subject, according to embodiments of the technology described herein. The machine learning model architt includes the time point prediction model 204 described herein with reference to FIGs the response prediction model 302 described herein with reference to FIG. 3. The tim prediction model 204 includes parameters (e.g., neural network weights) trained to pi of features generated from datasets 212 to output time points time points 216A, 216B 216D, 216E. As shown in FIG. 4B, the time point 216C is filtered out from the set ofThe response prediction model 302 includes parameters trained to process sets of feat generated from dataset collections 310A, 310B, 310D, 310E to output predicted inter responses 314A, 314B, 314D, 314E.

[0107] FIG. 5A is an example process 500 of extracting intervention responses fc from data associated with the subject, according to some embodiments of the technol described herein. In some embodiments, process 500 may be performed by interventi extraction system 100 described herein with reference to FIGs. 1A-1C.

[0108] Process 500 begins at block 502, where the system accesses imaging relate (e.g., radiology report documents) associated with a subject. In some embodiments, tl may access the imaging related data associated with the subject from a database (e.g., 120 described herein with reference to FIG. 1 A) storing data associated with subjects database may be a corpus of compiled subject data. For example, the system may acc imaging related data associated with the subject from a database storing clinical data with subjects obtained from an EHR database.

[0109] In some embodiments, the system may access imaging related data genera particular time period. For example, the system may access imaging related data gene time period following diagnosis of a condition (e.g., diagnosis of a tumor in the subje another example, the system may access imaging related data generated in a time per range of dates) specified by a user. In some embodiments, the system may identify in related data generated in a given time period based on a timestamp associated with th related data. For example, the system may identify radiology report document(s) gene given time period by determining whether timestamps associated with the radiology i document(s) are within the time period.

[0110] Next, process 500 proceeds to block 504, where the system divides the im related data accessed at block 502 into multiple partitions. In some embodiments, the may divide the imaging related data into partitions associated with respective time pe example, each partition may store imaging related data generated in a particular timethe multiple partitions, which each partition is associated with a respective time perio data partition may include radiology report document(s) generated in a time period as with the data partition. Example data partitioning techniques that may be performed z described herein with reference to the data partitioning module 102 of FIGs. 1A-1C.

[0111] In some embodiments, the system may include data in addition to imaging data in a given data partition. The system may include, in a data partition, clinical vis documents generated during a time period associated with the data partition. The add may further be used in determining a time point. For example, the system may extrac clinical visit note documents generated in a time period associated with a data partitic include the extracted text in the data partition.

[0112] Next, process 500 proceeds to block 506, where the system uses data frorr partitions to determine time points (e.g., dates) indicating predicted time periods of ir response determinations (e.g., performed by a physician). The system may determine point for each of the data partitions. In some embodiments, the system may determine point using a data partition by: (1) using data from the data partition to generate a set and (2) processing the set of features using a machine learning model to obtain a time example process for determining a time point using a data partition is described herei reference to FIG. 5B. FIG. 2 A illustrates an example use of data from data partitions determine time points indicating predicted time periods of intervention response dete: In some embodiments, the time points determined at block 506 may be used to extrac intervention responses from data associated with the subject (e.g., by performing proc described herein with reference to FIG. 7).

[0113] FIG. 5B is an example process 550 of using data from a data partition stor related data to determine a time point, according to some embodiments of the technol described herein. In some embodiments, process 550 may be performed by interventi extraction system 100 described herein with reference to FIGs. 1A-1C. In some embc process 550 may be performed at block 506 of process 500 described herein with reftexample, the system may extract text snippets from radiology report document(s) and visit note document(s) included in the data partition. In some embodiments, the syste extract text snippets from the data in the partition by: (1) searching for one or more k data (e.g., document(s)) of the partition; and (2) extracting snippet(s) of text surround identified keyword(s) in the data. For example, the system may extract a snippet of te certain number of characters or words following and / or preceding an identified keyw< some embodiments, the system may extract data from the partition by extracting valu more database fields. For example, the system may access values of metadata fields f documents included in the data partition.

[0115] Next, process 550 proceeds to block 554, where the system generates a sei using the data extracted from the data partition. In some embodiments, the system rm an embedding of data extracted from the data partition. The embedding may comprisi values (e.g., real number values) representing the extracted data (e.g., extracted text), embedding may be one or more vectors or matrices of real number values representin extracted data. For example, the extracted text may include a text snippet. The systen tokenize the text snippet to obtain a token vector. For example, each value in the toke may be one or more words from the text snippet. The system may generate an embed each token of the token vector (e.g., by determining a vector of 128 real number valu< representing the token). The system may combine the embeddings generated for the t set of features representing the extracted text. As another example, the extracted data include a value of a field (e.g., a metadata field). The system may generate an embed< representing the field value. The system may combine the embedding representing th with one or more embeddings representing extracted text to obtain the set of features.

[0116] Next, process 550 proceeds to block 556, where the system processes the : features using a trained machine learning model to obtain a time point indicating a pr period in which an intervention response was determined for the subject. For exampk trained machine learning model may be time point prediction model 204 described he(e.g., weights) of layers of the model to determine an output. In some embodiments, t machine learning model may output a classification result. The classification result m values (e.g., a probability values) each associated with a respective one of multiple p< points (e.g., dates in a year). The system may select a time point with the highest vah time point indicative of the intervention response determination for the subject.

[0117] FIG. 6 is another example process 600 of identifying dataset collections fr to extract intervention responses, according to some embodiments of the technology < herein. In some embodiments, process 600 may be performed by intervention respon extraction system 100 described herein with reference to FIGs. 1A and ID.

[0118] Process 600 begins at block 602, where the system accesses imaging relate associated with a subject. The system may access imaging related data as described a described herein with reference to FIG. 5B.

[0119] Next, process 600 proceeds to blocks 604-606 where the system processes imaging related data using a trained machine learning model to obtain time points inc respective predicted time periods of intervention response determination for a subject 604, the system generates sets of features using data from the imaging related data. Ir embodiments, the system may generate the sets of features by extracting text from th< related data and using the extracted text to generate the sets of features. The system n numeric representations of the text (e.g., numeric vectors) that can be provided as inp trained machine learning model.

[0120] In some embodiments, the system may generate a set of features correspoi each dataset (e.g., imaging report document) in the imaging related data. The system generate an embedding of text from a dataset. The embedding may comprise numeric (e.g., real number values) representing the text. The embedding may be one or more matrices of real number values representing the extracted text. For example, the extra may include a text snippet. The system may tokenize the text snippet to obtain a tokei where each value in the token vector may be one or more words from the text snippetfield). The system may generate an embedding representing the field value. The syste combine the embedding representing the field value with one or more embeddings re extracted text to obtain the set of features.

[0121] Next, at block 606, the system processes the sets of features using a traine learning model to obtain time points. The time points may indicate predicted time pei which an intervention response was determined for the subject. For example, the trair learning model may be time point prediction model 204 described herein with referer 2A-2B. The system may process the sets of features using parameters of the time poii model (e.g., accessed from memory of the system). For example, the trained machine model may be a neural network model and the sets of features may be provided as in] input layer of the neural network model. The system may use parameters (e.g., weigh of the model to determine outputs. In some embodiments, the trained machine learnir may output classification results for each of the sets of features. The classification res each include values (e.g., a probability values) associated with respective multiple po points (e.g., dates in a year). The system may select a time point with the highest vah time point determined for a respective set of features.

[0122] Next, process 600 proceeds to block 608, where the system filters the time obtained at block 606 to obtain a subset of time points. In some embodiments, the sys filter the time points by generating a filtered set of time points in which each pair of t is separated by at least a threshold amount of time (e.g., two weeks). The system may time points by: (1) identifying any time points that are less than the threshold amount after a preceding time point; and (2) remove any identified time points to obtain the f of time points. In some cases, the filtered set of time points may be a subset of the tin (e.g., because one or more of the time points were less than the threshold amount of t respective preceding time point(s)). In some cases, the filtered set of time points may the time points obtained at block 606 (e.g., because all pairs of time points were sepa least the threshold amount of time). In some embodiments, the filtered set of time poidescribed herein. In some embodiments, process 700 may be performed by interventi extraction system 100 described herein with reference to FIGs. 1A-1D.

[0124] Process 700 begins at block 702, where the system obtains time points ind predicted time periods in which an intervention response was determined for the subj example, the time points may be obtained by performing processes 500, 550 describe with reference to FIG. 5B or process 600 described herein with reference to FIG. 6.

[0125] Next, process 700 proceeds to block 704, where the system generates data collections using the time points. Each of the dataset collections may include one or r example, the system may identify a collection of clinical visit note document(s) using point. In some embodiments, the system may generate a dataset collection correspond time point. The system may generate a dataset collection corresponding to a time poii identifying one or more datasets (e.g., clinical note documents) generated after the tir and (2) store the identified dataset(s) in the dataset collection. In some embodiments, may identify a particular number (e.g., three) of earliest generated datasets after the ti For example, the system may identify the three earliest oncology clinical visit note d( generated after the time point to use in the determination of an intervention response, embodiments, the system may identify dataset(s) generated within a threshold amoun (e.g., 1 week, 2 weeks, 3 weeks, or another suitable amount of time) after the time po

[0126] Next, process 700 proceeds to blocks 706-710 where the system processes collections using a trained machine learning model to obtain intervention responses, i 706, the system extracts data from the dataset collections. In some embodiments, the extract data from a dataset collection by extracting text snippets from dataset(s) in the For example, the system may extract text snippets from clinical visit note document^ in the collection. In some embodiments, the system may extract text snippets from th< collection by: (1) searching for one or more keywords in datasets (e.g., document(s)) collection; and (2) extracting snippet(s) of text surrounding identified keyword(s) in t example, the system may extract a snippet of text within a certain number of characte

[0127] Next, process 700 proceeds to block 708, where the system generates sets using the data extracted from the dataset collection. In some embodiments, the systen generate embeddings of number values (e.g., real number values) representing the exi The embeddings may each be one or more vectors or matrices of real number values : data extracted from a respective data collection. For example, extracted data may incl snippet. The system may tokenize the text snippet to obtain a token vector. For exam] value in the token vector may be one or more words from the text snippet. The systen generate an embedding for each token of the token vector (e.g., by determining a veci real number values representing the token). The system may combine the embedding: for the tokens as a set of features representing the extracted text. As another example, extracted data may include a value of a field (e.g., a metadata field). The system may embedding representing the field value. The system may combine the embedding rep the field value may with one or more embeddings representing extracted text to obtai features.

[0128] In some embodiments, the system may apply filtering to the extracted set < part of generating the set of features. The system may filter data that is similar or idei data in a previously processed dataset collection. For example, the system may filter < snippets that are identical or sufficiently similar to text snippets extracted from a prey processed dataset collection. An example process of filtering the data is described hei reference to FIG. 8.

[0129] In some embodiments, the system may generate a set of features using dat dataset collection by determining a previously determined intervention response and indication of the previously determined intervention response in the set of features. F the system may determine an intervention response determined (e.g., by performing ]' based on a time point immediately preceding the time point obtained at block 702. Tt may include an indication of the intervention response in the set of features. For exan previously determined intervention response may be a classification into one of multi

[0130] After the generation of the set of features at block 708, process 700 procet 710, where the system processes the sets of features using an intervention response pi model to obtain intervention responses. For example, the intervention response predic may be the intervention response prediction model 302 described herein with referen* The system may process the sets of features using parameters of the intervention resp prediction model (e.g., accessed from memory of the system). For example, the inter' response prediction model may be a neural network model and the set of features ma; provided as input to an input layer of the neural network model. The system may use (e.g., weights) of layers of the model to determine an output for a set of features. In s< embodiments, the intervention response prediction model may output a classification classification result may be a value (e.g., a probability value) associated with each of possible intervention response categories (e.g., CR, PR, CD, SD, and unknown). The select a response category with the highest value to be the predicted intervention resp

[0131] FIG. 8 is an example process 800 of filtering data extracted from a dataset (e.g., a collection of clinical documents), according to some embodiments of the tech described herein. In some embodiments, process 800 may be performed by interventi extraction system 100 described herein with reference to FIGs. 1A-1D. In some emb< process 800 may be performed at block 708 of process 700 described herein with reft FIG. 7.

[0132] Process 800 begins at block 802, where the system accesses text snippets < from a dataset collection currently being processed and one or more preceding datase last 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10 datasets prior to the dataset collection). Text snippets been extracted from datasets as described herein at block 706 of process 700.

[0133] Next, process 800 proceeds to block 804, where the system determines a r similarity between text snippet(s) extracted from the dataset collection and the text su extracted from the preceding dataset(s). In some embodiments, the system may deteri measure of similarity between a pair of text snippets by determining a degree of over

[0134] Next, process 800 proceeds to block 806, where the system filters out text that meet a threshold level of similarity with a text snippet extracted from the precedi dataset(s). In some embodiments, the system may filter out text snippets that have a u degree of overlap with a text snippet extracted from the preceding dataset(s). For exai system may filter out text snippets that have at least an 80% overlap with a text snipp from the preceding dataset(s). The system may thus eliminate data related to an inters response associated with the preceding dataset(s) in extracting an intervention respon current dataset collection.

[0135] Next, process 800 proceeds to block 808, where the system uses the filtere snippets to generate a set of features. The system may generate a set of features as de block 706 of process 700.

[0136] FIG. 9 is a diagram 900 depicting filtering of data for a collection of datas according to some embodiments of the technology described herein. The filtering dej diagram 900 may be performed as part of process 800 described herein with reference (e.g., by intervention response extraction system 100 described herein with reference 1A-1C). In the example of FIG. 9, filtering is being performed for a collection OCV i documents determined based on a time point (referred to as “assessment date”). A 1 th< example of FIG. 9 filters OCV note documents, some embodiments may be configure perform filtering for other types of datasets in addition to or instead of OCV note doc

[0137] In the first step 902 of the filtering, OCV note documents prior to a cutoff loaded by the system. In the second step 904, the system identifies a collection of OC documents generated based an assessment date. For example, the assessment date ma timepoint identified by performing process 600 described herein with reference to FI( the example of FIG. 9, the system identifies a collection of the three earliest OCV noi documents generated after the assessment date. In the second step 904, the system rei note documents generated after the three earliest OCV note documents.

[0138] In the third step 906, the system extracts text snippets from the collectionhave a threshold similarity to text snippets that appeared in the preceding OCV note < to obtain a filtered set of text snippets extracted from the collection of OCT note doci the sixth step 912, the system keeps the filtered set of text snippets extracted from the of OCV note documents (e.g., for use in generating a set of features to determine an i response).

[0139] By performing the filtering depicted in FIG. 9, the system may eliminate t collection of OCV notes that is repeated from previous OCV note documents. This m the system to use text from the collection of OCV notes that is new (e.g., generated al assessment date) relative to the previous OCV note documents. The system may thus the system is not using previously authored text that may not relate to an intervention determined after the assessment date.

[0140] FIG. 10 shows a block diagram of an example computer system 1000 that used to implement some embodiments of the technology described herein. The comf device 1000 may include one or more computer hardware processors 1002 and non-ti computer-readable storage media (e.g., memory 1004 and one or more non-volatile si devices 1006). The processor(s) 1002 may control writing data to and reading data fr< memory 1004; and (2) the non-volatile storage device(s) 1006. To perform any of th< functionality described herein, the processor(s) 1002 may execute one or more procef executable instructions stored in one or more non-transitory computer-readable storaj (e.g., the memory 1004), which may serve as non-transitory computer-readable storaj storing processor-executable instructions for execution by the processor(s) 1002.

[0141] Having thus described several aspects of at least one embodiment of the te described herein, it is to be appreciated that various alterations, modifications, and irr will readily occur to those skilled in the art.

[0142] Such alterations, modifications, and improvements are intended to be part disclosure, and are intended to be within the spirit and scope of disclosure. Further, tl advantages of the technology described herein are indicated, it should be appreciated

[0143] The above-described embodiments of the technology described herein can implemented in any of numerous ways. For example, the embodiments may be imple using hardware, software or a combination thereof. When implemented in software, t code can be executed on any suitable processor or collection of processors, whether ]’ single computer or distributed among multiple computers. Such processors may be in as integrated circuits, with one or more processors in an integrated circuit component commercially available integrated circuit components known in the art by names sue! chips, GPU chips, microprocessor, microcontroller, or co-processor. Alternatively, a may be implemented in custom circuitry, such as an ASIC, or semicustom circuitry n from configuring a programmable logic device. As yet a further alternative, a process portion of a larger circuit or semiconductor device, whether commercially available, : custom or custom. As a specific example, some commercially available microprocess multiple cores such that one or a subset of those cores may constitute a processor. Ho processor may be implemented using circuitry in any suitable format.

[0144] Further, it should be appreciated that a computer may be embodied in any number of forms, such as a rack-mounted computer, a desktop computer, a laptop coi tablet computer. Additionally, a computer may be embedded in a device not generally as a computer but with suitable processing capabilities, including a Personal Digital 7 (PDA), a smart phone or any other suitable portable or fixed electronic device.

[0145] Also, a computer may have one or more input and output devices. These c be used, among other things, to present a user interface. Examples of output devices t used to provide a user interface include printers or display screens for visual presenta output and speakers or other sound generating devices for audible presentation of out Examples of input devices that can be used for a user interface include keyboards, an devices, such as mice, touch pads, and digitizing tablets. As another example, a comp receive input information through speech recognition or in other audible format.

[0146] Such computers may be interconnected by one or more networks in any si

[0147] Also, the various methods or processes outlined herein may be coded as s< is executable on one or more processors that employ any one of a variety of operating platforms. Additionally, such software may be written using any of a number of suita programming languages and / or programming or scripting tools, and also may be com executable machine language code or intermediate code that is executed on a framew virtual machine.

[0148] In this respect, aspects of the technology described herein may be embodii computer readable storage medium (or multiple computer readable media) (e.g., a coi memory, one or more floppy discs, compact discs (CD), optical discs, digital video d: magnetic tapes, flash memories, circuit configurations in Field Programmable Gate A other semiconductor devices, or other tangible computer storage medium) encoded w more programs that, when executed on one or more computers or other processors, pe methods that implement the various embodiments described above. As is apparent fr< foregoing examples, a computer readable storage medium may retain information for time to provide computer-executable instructions in a non-transitory form. Such a coi readable storage medium or media can be transportable, such that the program or pro; stored thereon can be loaded onto one or more different computers or other processor implement various aspects of the technology as described above. As used herein, the term "computer-readable storage medium" encompasses only a non-transitory compu medium that can be considered to be a manufacture (i.e., article of manufacture) or a Alternatively or additionally, aspects of the technology described herein may be embi computer readable medium other than a computer-readable storage medium, such as ; propagating signal.

[0149] The terms “program” or “software” are used herein in a generic sense to r< type of computer code or set of computer-executable instructions that can be employe program a computer or other processor to implement various aspects of the technolog described above. Additionally, it should be appreciated that according to one aspect c

[0150] Computer-executable instructions may be in many forms, such as progran executed by one or more computers or other devices. Generally, program modules in< routines, programs, objects, components, data structures, etc. that perform particular t implement particular abstract data types. Typically, the functionality of the program i may be combined or distributed as desired in various embodiments.

[0151] Also, data structures may be stored in computer-readable media in any sui For simplicity of illustration, data structures may be shown to have fields that are reh location in the data structure. Such relationships may likewise be achieved by assigni for the fields with locations in a computer-readable medium that conveys relationship the fields. However, any suitable mechanism may be used to establish a relationship 1 information in fields of a data structure, including through the use of pointers, tags or mechanisms that establish relationship between data elements.

[0152] Various aspects of the technology described herein may be used alone, in combination, or in a variety of arrangements not specifically described in the embodi described in the foregoing and is therefore not limited in its application to the details arrangement of components set forth in the foregoing description or illustrated in the For example, aspects described in one embodiment may be combined in any manner described in other embodiments.

[0153] Also, the technology described herein may be embodied as a method, of v examples are provided herein including with reference to FIGs. 3 and 7. The acts perl part of any of the methods may be ordered in any suitable way. Accordingly, embodi be constructed in which acts are performed in an order different than illustrated, whic include performing some acts simultaneously, even though shown as sequential acts i illustrative embodiments.

[0154] Further, some actions are described as taken by an “actor” or a “user”. It s appreciated that an “actor” or a “user” need not be a single individual, and that in son

[0155] Use of ordinal terms such as “first,” “second,” “third,” etc., in the claims t claim element does not by itself connote any priority, precedence, or order of one cla: over another or the temporal order in which acts of a method are performed, but are u as labels to distinguish one claim element having a certain name from another elemer same name (but for use of the ordinal term) to distinguish the claim elements.

[0156] Also, the phraseology and terminology used herein is for the purpose of d< and should not be regarded as limiting. The use of "including," "comprising," or "hav “containing,” “involving,” and variations thereof herein, is meant to encompass the it thereafter and equivalents thereof as well as additional items.

Claims

What is claimed is:CLAIMS1. A computer system for automatically extracting intervention responses from c associated with subjects, the computer system comprising: at least one processor; and a plurality of modules executed by the at least one processor, the plurality of i comprising a time point extraction module, a response data identification module, an< extraction module, wherein: the time point extraction module is configured to: access imaging related data associated with a subject; process the imaging related data using a first trained machine 1 (ML) model to obtain a plurality of time points, the processing compri generate a plurality of sets of features using data from t related data; and process the plurality of sets of features using the first tr model to obtain the plurality of time points; the response data identification module is configured to: generate, using the data associated with the subject, a dataset c each of at least some of the plurality of time points to obtain a pluralit; collections, the generating comprising: identify, in the data associated with the subject, one or datasets generated after the time point; and include the one or more datasets in the dataset collectio the response extraction module is configured to: process the plurality of dataset collections using a second train model to obtain a plurality of intervention responses, the processing c(process the set of features using the second trained ML obtain an intervention response of the plurality of intervention2. The system of claim 1 , wherein the first trained ML model is a first neural net model, and the second trained ML model is a second neural network model.

3. The system of claim 1, wherein the response extraction module is further coni generate a dataset storing the at least some time points and corresponding interventio: obtained using dataset collections generated using the at least some time points.

4. The system of claim 3, wherein generating a dataset storing the at least some I and corresponding intervention responses obtained using dataset collections generatei at least some time points comprises storing, in the dataset: a first time point of the at least some time points; a first intervention response corresponding to the first time point, the first inte response obtained using data from a first one of the plurality of dataset collections, th dataset collection comprising one or more datasets generated after the first time point a second time point of the at least some time points, the second time point suf the first time point; and a second intervention response corresponding to the second time point, the se< intervention response obtained using data from a second one of the plurality of datase collections, the second dataset collection comprising one or more datasets generated ; second time point.

5. The system of claim 1, wherein the plurality of time points comprises a plural6. The system of claim 1 , wherein each of at least some of the plurality of interv7. The system of claim 1, wherein the at least some time points are a subset of tf of time points and the time point extraction module is further configured to: identify the subset of time points such that each pair of the subset of time poir separated by at least a threshold amount of time.

8. The system of claim 7, wherein identifying the subset of time points comprise identifying one or more of the plurality of time points that are less than the thi amount of time after a respective preceding time point of the plurality of time points; filtering out the one or more time points from the plurality of time points to ol subset of time points.

9. The system of claim 7, wherein the threshold amount of time is two weeks.

10. The system of claim 1, wherein the plurality of modules further comprises a d partitioning module configured to: divide the imaging related data into a plurality of partitions; wherein generating the plurality of sets of features using the data from the im; data comprises using each of the plurality of partitions to generate a respective one ol plurality of sets of features.

11. The system of claim 10, wherein: the imaging related data comprises a plurality of imaging report documents; the plurality of partitions are associated with a respective plurality of time per dividing the imaging related data into the plurality of partitions comprises: dividing the plurality of imaging report documents into the plurality ol by storing, in each of the plurality of partitions, one or more of the plurality o: report documents generated in a respective time period associated with the pa13. The system of claim 12, wherein identifying the one or more earliest generate generated after the time point comprises identifying three earliest generated datasets ; time point.

14. The system of claim 1, wherein the at least one processor is further configure< the first trained ML model by performing training using training data comprising: sets of features generated using imaging related data associated with a pluralit subjects; and time point labels indicating target time point predictions for the sets of feature15. The system of claim 1, wherein the at least one processor is configured to obt; second trained ML model by performing training using training data comprising: sets of features generated from datasets associated with a plurality of subjects intervention response labels indicating target intervention response prediction sets of features.

16. The system of claim 1, wherein generating the plurality of sets of features usii from the image related data comprises generating the plurality of sets of features usin extracted from the image related data.

17. The system of claim 1, wherein generating the set of features using data from collection comprises: extracting text from the dataset collection; and generating the set of features using the text extracted from the dataset collect!18. The system of claim 17, wherein the text extracted from the dataset collectiondetermine that the measure of similarity meets a threshold level of similarity; remove the first set of text from the text extracted from the dataset collection measure of similarity meets the threshold similarity to obtain a filtered set of text; wherein generating the set of features using the text extracted from the datasei comprises generating the set of features using the filtered set of text.

19. A method for automatically extracting intervention responses from data assoc subjects, the method comprising: using at least one processor to perform: accessing imaging related data associated with a subject; processing the imaging related data using a first trained machine learn model to obtain a plurality of time points, the processing comprising: generating a plurality of sets of features using data from the im data; processing the plurality of sets of features using the first trainei to obtain the plurality of time points; identifying, from the data associated with the subject, a first co one or more datasets generated after a first one of the plurality of time generating, using the data associated with the subject, a dataset collect of at least some of the plurality of time points to obtain a plurality of dataset c the generating comprising: identifying, in the data associated with the subject, one or more generated after the time point; and including the one or more datasets in the dataset collection; an< processing the plurality of dataset collections using a second trained A obtain a plurality of intervention responses, the processing comprising, for ea< least some of the plurality of dataset collections: peneratinp a set of features nsinp data from the dataset collectii20. A non-transitory computer-readable medium storing instructions that, when e: at least one processor, cause the at least one processor to perform a method for autom extracting intervention responses from data associated with subjects, the method com accessing imaging related data associated with a subject; processing the imaging related data using a first trained machine learning (MI obtain a plurality of time points, the processing comprising: generating a plurality of sets of features using data from the image rel; processing the plurality of sets of features using the first trained ML rr obtain the plurality of time points; identifying, from the data associated with the subject, a first collection more datasets generated after a first one of the plurality of time points; generating, using the data associated with the subject, a dataset collection for least some of the plurality of time points to obtain a plurality of dataset collections, tl generating comprising: identifying, in the data associated with the subject, one or more datase after the time point; and including the one or more datasets in the dataset collection; and processing the plurality of dataset collections using a second trained ML mod a plurality of intervention responses, the processing comprising, for each of at least s< plurality of dataset collections: generating a set of features using data from the dataset collection; and processing the set of features using the second trained ML model to ol intervention response of the plurality of intervention responses.

Images