Phenotyping of clinical notes using natural language processing models

Abstract

This invention provides a system and method for phenotyping clinical data. The method includes obtaining episodic records containing unstructured clinical data from an electronic medical record (EMR) or electronic health record (EHR) relating to a patient. The method also includes filtering the episodic records by language pattern recognition to identify those that each contain representations related to a clinical state. The method also includes splitting each episodic record to obtain snippets containing tokens. The method also includes using a trained classifier to predict whether an episodic record represents an instance of a clinical state. The trained classifier includes an aggregate function that aggregates the snippets and outputs corresponding representations for each episodic record, and an interpret function that interprets the corresponding representations and outputs corresponding predictions about whether an episodic record represents an instance of a clinical state.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 63 / 420,466, filed October 28, 2022, which is hereby expressly incorporated by reference in its entirety for all purposes.

[0002] This application is directed to phenotyping clinical notes using natural language processing models.

Background Art

[0003] Patient records, such as electronic health records (EHRs) or handwritten records that can be digitized later to generate electronic medical records (EMRs), record the interactions between patients and healthcare providers. EHRs and EMRs are then stored in electronic medical systems curated for healthcare providers. These EHRs and EMRs typically include structured data containing medical codes used by healthcare providers for billing purposes, and unstructured data including clinical notes and findings made by physicians, physician assistants, nurses, and other personnel during patient examinations.

[0004] Together, EHRs and EMRs theoretically hold an enormous amount of clinical data that could be harnessed for the greater public health good. For example, the CDC estimated in 2019 that nearly 90% of office - based physicians used EHR or EMR systems to track patient care. Weighted estimates from the 2019 National Electronic Health Records Survey national public use file, CDC / National Center for Health Statistics. Using such rich clinical data, models could be generated for predicting disease risk, treatment outcomes, recommending personalized treatments, predicting disease - free survival after treatment, predicting disease recurrence, and so on.

[0005] However, in order to make this data available for model training, each electronic record needs to be appropriately labeled with one or more clinical phenotypes that the record holds data for. Traditionally, this has been done using either or both of the following methods: (i) computer-implemented rule-based models that evaluate medical codes within the structured data portion of electronic records, and (ii) manual chart review. However, these methods are considerably underperforming. Specifically, EHR and EMR systems are not standardized across the healthcare industry, meaning that data is presented differently across numerous record systems within the industry, and for this reason alone, traditional rule-based models are underperforming. Furthermore, these models cannot account for the use of inconsistent medical codes across different medical practices and healthcare providers, thereby preventing rules from generalizing across different EHR and EMR systems, as well as different healthcare providers using different coding practices. On the other hand, manual review is extremely cumbersome and time-consuming. A manual review of a single health record typically takes 30-60 minutes, but performing this across tens of millions, hundreds of millions, or billions of electronic health records is extremely slow and costly. Furthermore, manual reviews are also subject to the reviewer's bias. [Overview of the project]

[0006] Given the above background, what is needed in the art is an improved method and system for phenotyping electronic health records at an appropriate scale. This disclosure addresses these and other issues by using natural language processing to evaluate clinical notes contained in the unstructured portions of electronic health records with respect to relevant phenotypes. The disclosed system and method will both improve the performance of phenotyping electronic health records and facilitate the scaling of such phenotyping across large amounts of clinical data to enhance generalizability.

[0007] Accordingly, one aspect of the present disclosure provides phenotyping of clinical notes. In some embodiments, the method includes acquiring multiple episodic records in electronic format, wherein each individual episodic record within the multiple episodic records contains corresponding unstructured clinical data from electronic medical records (EMRs) or electronic health records (EHRs) relating to individual patients of multiple patients.

[0008] In some embodiments, the method includes obtaining corresponding unstructured clinical data from multiple medical assessments stored in an EMR or EHR for an individual patient for each episode record within a plurality of episode records.

[0009] In some embodiments, the method includes selecting multiple medical assessments by clustering all or some of the medical assessments stored in the EMR or EHR for an individual patient to obtain one or more corresponding medical assessment clusters, and by aggregating unstructured clinical data corresponding to each of the individual medical assessments within each of the one or more corresponding medical assessment clusters to form individual episode records.

[0010] In some embodiments, the clustering is at least partially time-based clustering.

[0011] In some embodiments, the clustering is one-dimensional clustering.

[0012] In some embodiments, the method includes obtaining corresponding unstructured clinical data from a single medical assessment stored in an EMR or EHR for individual episode records within a plurality of episode records.

[0013] In some embodiments, each episode record within a set of multiple episode records does not include a corresponding structured clinical record from the EMR or EHR.

[0014] In some embodiments, the method includes filtering multiple episode records by language pattern recognition to identify a quasi-episode record in which each contains an expression related to a clinical condition in the corresponding unstructured clinical data.

[0015] In some embodiments, language pattern recognition includes identifying a quasi-multiple episode records by matching one or more regular expressions against corresponding unstructured clinical data for each individual episode record within a plurality of episode records.

[0016] In some embodiments, language pattern recognition includes a machine learning model trained to identify language associated with a clinical condition.

[0017] In some embodiments, the clinical condition is atrial fibrillation.

[0018] In some embodiments, the method includes dividing the corresponding unstructured clinical data into corresponding snippets for each individual episode record within a quasi-multiple episode record. Each individual snippet within the corresponding snippets includes a corresponding set of one or more tokens.

[0019] In some embodiments, the separation of corresponding unstructured clinical data is performed before filtering of multiple episode records.

[0020] In some embodiments, the splitting of corresponding unstructured clinical data is performed after filtering multiple episode records.

[0021] In some embodiments, each snippet within a corresponding group of snippets has approximately the same number of tokens.

[0022] In some embodiments, for each individual episode record within a quasi-multiple episode record, each individual snippet within a corresponding snippet has a corresponding number of tokens, which is within 25% of the corresponding number of tokens for each of the other individual snippets within the corresponding snippet.

[0023] In some embodiments, for individual episode records within a quasi-multiple episode records, the division of corresponding unstructured clinical data includes tokenizing the corresponding unstructured clinical data to obtain multiple tokens, segmenting the multiple tokens to obtain multiple segments, wherein each individual segment within the multiple segments has approximately the same number of tokens, ranking the individual segments within the multiple segments based on the token values ​​within each individual segment, and removing one or more individual segments from the multiple segments based on the ranking to generate a corresponding multiple snippet for the individual episode record.

[0024] In some embodiments, for individual episode records within a quasi-multiple episode records, segmenting the corresponding unstructured clinical data includes: segmenting the corresponding unstructured clinical data to obtain a plurality of segments, each of which individual segments contains an individual portion of the corresponding unstructured clinical data; tokenizing the individual portions of the corresponding unstructured clinical data within each individual segment to obtain a plurality of tokenized segments; segmenting individual tokenized segments in the plurality of tokenized segments that have a corresponding number of tokens exceeding a threshold to obtain a second plurality of tokenized segments; ranking the individual segments within the second plurality of tokenized segments based on the token values ​​within each individual tokenized segment; and removing one or more individual tokenized segments from the second plurality of tokenized segments based on the ranking to generate a plurality of corresponding snippets for the individual episode record.

[0025] In some embodiments, for each individual episode record within a plurality of episode records, splitting the corresponding unstructured clinical data involves segmenting the corresponding unstructured clinical data sentence by sentence to obtain a plurality of segments, where each individual segment within the plurality of segments contains an individual part of the corresponding unstructured clinical data; obtaining a plurality of segments; tokenizing each individual part of the corresponding unstructured clinical data in each individual segment within the plurality of segments to obtain a plurality of tokenized segments; splitting individual tokenized segments having a corresponding token count exceeding a first threshold of the number of tokens in the plurality of tokenized segments to obtain a second plurality of tokenized segments; merging individual tokenized segments having a corresponding token count below a second threshold of the number of tokens in the second plurality of tokenized segments to obtain a third plurality of tokenized segments; ranking individual segments within the third plurality of tokenized segments based on the values of the tokens within each individual tokenized segment; removing one or more individual tokenized segments from the third plurality of tokenized segments based on the ranking, thereby generating a corresponding plurality of snippets for each individual episode record.

[0026] In some embodiments, ranking is at least partially based on a scoring system that rewards the presence of tokens found in a list of token priorities.

[0027] In some embodiments, the scoring system penalizes the presence of tokens found in a list of token de-prioritizations.

[0028] In some embodiments, the corresponding plurality of snippets is a predetermined number of snippets.

[0029] In some embodiments, the method includes predicting, for each of the episode records in a plurality of quasi-episode records, whether an individual episode record represents an instance of a clinical state by inputting a plurality of corresponding snippets regarding the individual episode record into a classifier that includes a first part and a second part, the first part including an aggregation function that aggregates the plurality of corresponding snippets to output a corresponding representation regarding the individual episode record, and the second part interpreting the corresponding representation and outputting a corresponding prediction as to whether the individual episode record represents an instance of a clinical state.

[0030] In some embodiments, the first part of the classifier includes a multi-head encoder that outputs, for each of the individual snippets in the plurality of corresponding snippets regarding each of the individual episode records in a plurality of quasi-episode records, a corresponding contextualized token tensor for each of the individual tokens in a corresponding set of one or more tokens, thereby forming a plurality of corresponding contextualized token tensors regarding the individual snippets.

[0031] In some embodiments, the first part of the classifier further includes a multi-head intra-attention mechanism that aggregates, for each of the individual episode records in a plurality of quasi-episode records, the plurality of corresponding contextualized token tensors regarding each of the individual snippets in the plurality of corresponding snippets to output a corresponding contextualized snippet tensor, thereby forming a plurality of corresponding contextualized snippet tensors regarding the individual episode records.

[0032] In some embodiments, the first part of the classifier further includes an inter-attention mechanism that aggregates the plurality of corresponding contextualized snippet tensors regarding each of the individual episode records in a plurality of quasi-episode records to output a contextualized tensor of the episode record corresponding to the individual episode record.

[0033] In some embodiments, the second part of the classifier includes a model that, for each individual episode record within a quasi-multiple episode records, outputs a corresponding prediction about whether the individual episode record represents an instance of a clinical condition, in response to the input of a corresponding representation for the individual episode record into the model.

[0034] In some embodiments, the second part of the classifier includes a model selected from the group consisting of neural networks, support vector machines, naive Bayes algorithms, nearest neighbor algorithms, boost tree algorithms, random forest algorithms, convolutional neural networks, decision trees, regression algorithms, and clustering algorithms.

[0035] In some embodiments, the second part of the classifier includes a linear transformation that, for each episode record in a quasi-multiple episode records, converts the individual outputs of the first part of the classifier into corresponding scalar numbers compared to a threshold to output the corresponding prediction.

[0036] In some embodiments, the linear transformation is an affine transformation.

[0037] In some embodiments, the classifier includes at least 500 parameters, at least 1,000 parameters, at least 5,000 parameters, at least 10,000 parameters, at least 50,000 parameters, at least 100,000 parameters, at least 250,000 parameters, at least 500,000 parameters, at least 1,000,000 parameters, at least 10M parameters, at least 100M parameters, at least 1MM parameters, at least 10MM parameters, or at least 100MM parameters.

[0038] In some embodiments, the method includes labeling each individual episode record in a quasi-multiple episode records that is expected to represent an instance of a clinical condition, thereby forming a set of episode records, where each individual episode record in the set of episode records represents an instance of a clinical condition.

[0039] In some embodiments, the method includes training a model to predict clinical outcome using a set of episode recordings.

[0040] Another aspect of this disclosure provides a computer system for phenotyping clinical notes. The computer system comprises one or more processors and one or more memory addressable by the processors. The memory stores at least one program to be executed by one or more processors. The at least one program includes instructions for performing any of the methods described herein.

[0041] Another aspect of this disclosure provides a non-temporary computer-readable storage medium. When executed by a computer system, the non-temporary computer-readable storage medium stores instructions that cause the computer system to perform any of the methods described herein. [Brief explanation of the drawing]

[0042] The drawings illustrate examples of the systems and methods of this disclosure. It should be expressly understood that the descriptions and drawings are for illustrative purposes only and aid in understanding, and are not intended to limit the systems and methods of this disclosure.

[0043] [Figure 1] Figure 1 shows a computer system according to some embodiments of the present disclosure.

[0044] [Figure 2]Figure 2 shows a schematic diagram of a system for phenotyping clinical data according to some embodiments of this disclosure.

[0045] [Figure 3] Figure 3 shows a comparative example of different techniques for phenotyping clinical notes according to some embodiments of the present disclosure.

[0046] [Figure 4A] Figures 4A, 4B, and 4C illustrate exemplary methods for segmenting or dividing text according to certain embodiments of the present disclosure. [Figure 4B] Same as above. [Figure 4C] Same as above.

[0047] [Figure 5A] Figure 5A is a schematic diagram of an exemplary mechanism according to some embodiments of the present disclosure.

[0048] [Figure 5B] Figure 5B shows an exemplary architecture having a snippet encoder according to some embodiments of the present disclosure.

[0049] [Figure 5C] Figure 5C shows an exemplary architecture having a concept encoder according to some embodiments of the present disclosure.

[0050] [Figure 6] Figure 6 shows a schematic diagram of an exemplary training flow according to some embodiments of the present disclosure.

[0051] [Figure 7] Figure 7 shows an exemplary set of labels 700 used to train a classifier according to some embodiments of this disclosure.

[0052] [Figure 8A]Figures 8A–8G show flowcharts of exemplary methods for phenotyping clinical data according to certain embodiments of the present disclosure. [Figure 8B] Same as above. [Figure 8C] Same as above. [Figure 8D] Same as above. [Figure 8E] Same as above. [Figure 8F] Same as above. [Figure 8G] Same as above.

[0053] [Figure 9] Figure 9 shows the area under the precision-recall curve (AUPRC) of a validation set for hold-out episodes according to some embodiments of the present disclosure.

[0054] [Figure 10] Figure 10 shows the results of an interpretable model for a virtual text snippet according to some embodiments of the present disclosure.

[0055] Similar reference numbers refer to corresponding parts across several figures within a drawing. [Modes for carrying out the invention]

[0056] Here, embodiments are referenced in detail, the embodiments illustrated in the accompanying drawings. Numerous specific details are provided in the following detailed description to allow for a full understanding of the disclosure. However, it will be apparent to those skilled in the art that the disclosure can be practiced without these specific details. In other cases, well-known methods, procedures, components, circuits, and networks are not described in detail so as not to unnecessarily obscure the aspects of the embodiments.

[0057] This disclosure provides a system and method for phenotyping clinical notes. In some embodiments, a natural language processing (NLP) model is trained to detect the presence of a clinical condition (e.g., atrial fibrillation) using unstructured clinical notes by learning at scale from labels generated from validated structured EHR and invoice creation code definitions. Such methods facilitate scaling disease labeling methods across large amounts of clinical data without being plagued by discrepancies resulting from variability in coding practices.

[0058] Phenotypes correspond to a list of patient identifiers and diagnosis dates representing a diagnosis of a clinical condition, identified throughout the EHR. Typically, specialists perform chart reviews to determine whether records correspond to a disease diagnosis. This manual process is time-consuming and prone to errors. Therefore, an automated method for labeling the presence of disease in EHR data is desired. Conventional systems use labels generated from the definition of invoice creation codes (e.g., "at least two relevant ICD codes used within the last year"). While such labels may be accurate within one healthcare system, they cannot be generalized across systems due to variability in coding practices.

[0059] A phenotypic model is any set of rules or transformations that produce a phenotype as an output. This includes both manual rule-based approaches and machine learning models. Phenotypic models can be used to generate labels for risk prediction models that can predict the risk of a particular disease from clinical signals. Phenotypic models can also be used for population health monitoring and identifying a history of disease.

[0060] Conventional phenotyping models do not generalize to novel systems with different coding practices. Such models may be limited to relatively simple rule combinations that can lead to poor performance and / or be susceptible to bias (e.g., model bias stemming from expert bias regarding relevant diagnostic codes). Existing phenotypes for cardiovascular diseases are heavily reliant on codebase definitions, which often suffer from poor sensitivity and generalizability for low-prevalence diseases. High-quality EHR phenotypes and disease labels are essential for evidence generated from cohort studies and predictions from machine learning models.

[0061] Traditional phenotypic models ignore clinical notes, despite the fact that important signals (e.g., symptoms) are often present only in clinical notes. Clinical notes are typically lengthy; curators spend an average of 30 minutes reading and analyzing the event-level information in them. These notes are also low-density, meaning that much of this information is irrelevant. The meaning of any given clinical term depends on the context. Clinical terms can be confirmatory, retractive, past medical history, family history, suspected, or risk factor. Because much of the text is clinically concise, important terms can be expressed in many different ways. Conflicting information may exist due to developments in the clinical explanation and changes in diagnosis (particularly in differences in diagnosis).

[0062] In this specification, terms such as "first," "second," etc., may be used to describe various elements, but it will be understood that these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without departing from the scope of this disclosure, the first subject may be referred to as the second subject, and similarly, the second subject may be referred to as the first subject. The first and second subjects are both subjects, but they are not the same subject.

[0063] The terms used herein are for the sole purpose of describing specific embodiments and are not intended to be limitations of the invention. Where used in the description of the invention and in the appended claims, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context otherwise clearly indicates. Where used herein, the terms “and / or” refer to and encompass any possible combination of one or more of the enumerated items relating to the invention. Where used herein, the terms “comprises” and / or “comprising” specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0064] As used herein, the term “if” may, depending on the context, be interpreted as “when” or “upon,” “in response to detection,” or “in response to determination.” Similarly, the phrases “if determined” or “if detected [the described condition or event]” may, depending on the context, be interpreted as “at the time of determination,” or “in response to determination,” or “at the time of detection [of the described condition or event],” or “in response to detection [of the described condition or event].”

[0065] Figure 1 shows a computer system 100 for phenotyping clinical notes according to one embodiment. In a typical embodiment, the computer system 100 comprises one or more computers. In Figure 1, for illustrative purposes, the computer system 100 is represented as a single computer containing all the functions of the computer system 100 of this disclosure. However, the disclosure is not limited in this way. The functions of the computer system 100 can extend across any number of networked computers and / or reside on each of several networked computers and / or virtual machines. Those skilled in the art will understand that a wide range of different computer topologies are possible for the computer system 100, and that all such topologies are within the scope of this disclosure.

[0066] With the foregoing in mind, returning to Figure 1, the computer system 100 includes one or more processing units (CPUs) 59, a network or other communication interface 84, a user interface 78 (including, for example, an optional display 82 and an optional keyboard 80 or other form of input device), memory 92 (for example, random access memory, persistent memory, or a combination thereof), one or more magnetic disk storage and / or persistent devices 90 optionally accessed by one or more controllers 88, one or more communication buses 12 for interconnecting the aforementioned components, and a power supply 79 for supplying power to the aforementioned components. Data in memory 92 can be seamlessly shared with non-volatile memory 90 or portions of memory 92 that are non-volatile or persistent, using known computing techniques such as caches, to the extent that the components of memory 92 are not persistent. Memory 92 and / or memory 90 may include large-capacity storage located remotely from the central processing unit 59. In other words, some data stored in memory 92 and / or memory 90 may actually be hosted on a computer that is outside the computer system 100 but can be electronically accessed by the computer system 100 via the internet, intranet or other forms of network or electronic cable using the network interface 84. In some embodiments, the computer system 100 utilizes a model that runs from memory associated with one or more graphics processing units in order to improve the speed and performance of the system. In some alternative embodiments, the computer system 100 utilizes a model that runs from memory 92 rather than from memory associated with the graphics processing units.

[0067] The memory 92 of computer system 100 stores the following: • Operating system 34, which includes procedures for handling various basic system services. An input / output module 64 for electronically acquiring episode records containing corresponding unstructured clinical data from one or more electronic medical records (EMRs) or electronic health records (EHRs) relating to a patient. In some embodiments, the input / output module 64 labels episode records that are expected to represent instances of clinical conditions to form a set of episode records. In some embodiments, the input / output module 64 trains a model to predict outcomes of clinical conditions using the labeled episode records. • Clinical data including unstructured data and optionally structured data (e.g., invoice creation codes)36. Unstructured data may include unstructured clinical data from electronic medical records (EMRs) or electronic health records (EHRs) relating to patients. • Episode records 38, including unstructured clinical data from individual patient electronic medical records (EMRs) or electronic health records (EHRs) for multiple patients. A language pattern recognition module 40 filters the episode records 38 using language pattern recognition to identify episode records containing expressions related to the clinical state. In some embodiments, the language pattern recognition module 40 matches one or more regular expressions against the corresponding unstructured clinical data. In some embodiments, language pattern recognition includes a machine learning model trained to identify language related to the clinical state. • Expression 42 which may include regular expressions for use by the language pattern recognition module 40. Expression 42 may be optional in a system that uses a machine learning model for language pattern recognition. • A split module 44 containing snippets 46 and tokens 48. The split module 44 splits unstructured clinical data relating to an episode record into corresponding snippets. Each snippet contains a corresponding set of tokens, which may include lexical tokens such as words. The representations of individual tokens and snippets may contain vectors, sometimes called embeddings. An accumulation or concatenation of these vectors or embeddings constitutes a tensor. Snippets and tokens may also be referred to as tensors, because snippets and tokens are typically batched and concatenated during training. The classifier 50 includes an aggregation module 52 (also referred to as the first part of the classifier 50) and an interpretation module 54 (also referred to as the second part of the classifier 50). The first part includes an aggregation function that aggregates corresponding snippets with respect to an episode record and outputs a corresponding representation. The second part interprets the corresponding representation and outputs a corresponding prediction about whether the episode record represents an instance of a clinical condition. The aggregation module 52 and the interpretation module 54 include individual parameters (e.g., parameters obtained from training a machine learning model). • Optionally, a clustering module 56 clusters medical assessments stored in the patient's EMR or EHR to obtain medical assessment clusters. The clustering module 56 also aggregates unstructured clinical data corresponding to each medical assessment within each individual medical assessment cluster to form individual episode records. In some embodiments, clustering uses time-based clustering (e.g., based on the date of the medical assessment stored in the EMR or HER). In some embodiments, clustering is one-dimensional clustering. Optionally, a training module 58 containing labels 60 and a training dataset 52 for training a classifier 50.

[0068] In some implementations, one or more of the data elements or modules identified above in the computer system 100 are stored in one or more of the previously mentioned memory devices and correspond to a set of instructions for performing the functions described above. The data, modules, or programs (e.g., sets of instructions) identified above do not need to be implemented as separate software programs, procedures, or modules; therefore, various subsets of these modules can be combined or rearranged differently in various implementations. In some implementations, memories 92 and / or 90 optionally store a subset of the modules and data structures identified above. Furthermore, in some embodiments, memories 92 and / or 90 store additional modules and data structures not described above. Details of the modules and data structures identified above are further described below with reference to Figures 2-8.

[0069] Figure 2 shows a schematic diagram of a system 200 for phenotyping clinical data according to one embodiment. System 200 may be run using a computer system (for example, computer system 100 shown and described above with reference to Figure 1). System 200 is sometimes referred to as an extractor-classifier network.

[0070] Some embodiments preprocess clinical notes. Some embodiments use long clinical notes (e.g., about 100,000 words) to use a modern, pre-trained model with word count limitations (e.g., 512 words) without needing to discard context. In some embodiments, clinical notes are aggregated into episodes 202. A visit may include interactions between the patient and the healthcare provider who consequently records the clinical notes in the EHR system. Episodes 202 contain clusters of visits representing a single hospitalization. Typically, a single hospitalization may be recorded as multiple visits. In some embodiments, episode boundaries are determined on a patient-by-patient basis using one-dimensional clustering on visit dates (e.g., kernel density estimation (KDE)). In some embodiments, notes between boundaries are aggregated together.

[0071] Episode 202 is input to extractor 204 (e.g., language pattern recognition module 40) to obtain candidate episode 206 (sometimes called candidate). In some embodiments, extractor 204 uses regular expressions to filter the data. For example, the extractor may use the following regular expression for AFib (atrial fibrillation): (?i)atrial fibrillation|\Wafib\W|\saf\s|\Wa.fib\W|atrial flutter|aflutter| \Wa.flutter\W. In some embodiments, extractor 204 uses regular expressions to filter the set of clinical notes to only those that are likely to refer to a clinical condition for model decision-making.

[0072] Traditional machine learning models can take more than 0.1 seconds per episode for inference without any form of filtering. Extractor 204 significantly reduces training and estimation time, modifies the computational budget, and eliminates training-serving skew. Specifically, a very large number of notes can be reduced by at least an order of magnitude by being passed through the machine learning model, saving computational costs. Furthermore, the extractor increases the generalizability of the classifier by mitigating the effects of training-serving skew (the difference between model performance during training and performance during serving or inference). This helps to focus classifier 224 on the narrower task of distinguishing between positive mentions and non-primary mentions (e.g., "this patient has afib" vs. "the patient has a family history of afib"). In some experiments, extractor 204 showed a sensitivity of 92% and a positive predictive value (PPV) of 22%. The 92% sensitivity is a conservative estimate, and the estimate from the chart review pushed this sensitivity up to nearly 98%. The high recall ensures that the majority of positive cases are captured. Low precision is not a problem. Classifier 224 is trained to explicitly remove false positives from the candidate pool. The prevalence of the training data is the PPV of the extractor.

[0073] Figure 3 shows comparative examples 300 of different techniques for phenotyping clinical notes according to some embodiments. The objective is to predict whether a given episode contains an instance of the clinical condition Afib (atrial fibrillation). The total samples 302 contain three classes: no mention of AFib (shown in red), non-major mention of AFib (shown in blue), and AFib-positive mention (shown in green). Downsampling of negatives 304 results in some no mention of AFib, and stratified sampling of negatives 306 is not sufficient to narrow the boundaries between the three classes. On the other hand, an extractor-classifier network 200 using extractor 204 produces results 308 that distinguish between non-major AFib mentions and AFib-positive mentions.

[0074] Referring again to Figure 2, the text of an episode may be too large to feed into a deep learning model. Therefore, in some embodiments, the text (e.g., unstructured text corresponding to each episode) is segmented or divided into approximately equal snippets 226, taking sentence boundaries into account. In some embodiments, the text is ranked and trimmed according to several medically relevant words within each snippet. In some embodiments, the number of snippets and / or the number of words per snippet is limited (e.g., a maximum size of 512 snippets of 256 words each, totaling 131,072 words).

[0075] Figures 4A, 4B, and 4C illustrate exemplary methods for segmenting or splitting text according to some embodiments. Figure 4A shows an exemplary basic method 400 for splitting text. A database 404 stores raw episode text 402 of arbitrary length. This raw text is tokenized (406) to generate a list 408 of N tokens. This list is segmented (410) or split into M segments 412, each segment having a predetermined number of tokens (256 in this embodiment). The segments are ranked to obtain an ordered list 416 of the segments (414). The ordered list is then trimmed (418) to obtain a predetermined number of segments 420 (in this embodiment, there are 512 segments, each with 256 tokens) that can be stored in a snippet array 422. This method is not site-specific. For example, raw text can be obtained from any number of sites that contributed to (or aggregated from several EHR systems) the EHR data. Since the splitting step is independent of the phrase, a phrase containing the text "no AFib" may be split into two snippets, one containing the token corresponding to "no" and the other containing the token corresponding to "AFib". This method can be ranked (414) based on the number of tokens in the priority list and / or the number of tokens in the lower priority list.

[0076] Figure 4B shows another exemplary method 432 for segmentation according to some embodiments. This method performs a segmentation step 410 before the tokenization step 406. The raw text 402 is segmented (410) into segments of a predetermined segment length S i Text segments 414 are obtained, each containing a total of N segments. These segments are tokenized (406) and M×S iThe segments are obtained. These tokenized segments are split to avoid long snippets (424). In this example, some segments (e.g., segment [87,···,22]) are split into multiple segments. The resulting segments (L segments in this example, each with 256 tokens) are ranked (414) to obtain an ordered list of segments 450, which is then trimmed (418) to obtain a reduced number of snippets 430 that can be stored in a snippet array 422. This method is less likely to split phrases and is more likely to maintain a coherent thought together. However, this method requires curating reasonably generalizable rules for splitting and may result in more snippets being discarded, as some snippets may contain far fewer than 256 tokens. Some embodiments split the raw text into roughly equal snippets of a given size (e.g., 256 tokens). Some embodiments avoid snippets being cut off in the middle of a sentence by first cutting the text into sentences and then combining adjacent sentences to obtain approximately the same number of token snippets (e.g., 256 token snippets).

[0077] Figure 4C shows yet another exemplary method 434 for performing sentence-based splitting according to some embodiments. Raw text 402 is sentenced (436) to obtain N sentences 440, which are tokenized (406) to obtain M sets of tokens 442. Long snippets (any set within the M sets) are split to obtain L sets, each set having a predetermined number of tokens (256 in this example). Some embodiments may generate a warning to alert the user regarding long snippets. Several short snippets can be merged (438) to obtain a candidate set of snippets 446, which are ranked (414) to obtain an ordered list, and also trimmed (418) to obtain a set of trimmed snippets 448 to be stored in a snippet array 422. This exemplary method is similar to that shown in Figure 4A in that the method is also not field-specific. The method does not require any specific rules regarding splitting or merging other than those mentioned above, and is useful for generating token snippets in a predetermined number. However, since this method aggregates different phrases, it requires proper sentence construction.

[0078] In some embodiments, filtering of regular expressions is used to divide the raw text 402. An example of a regular expression syntax that may be used to divide the raw text into sentences is "r''". In some embodiments, certain punctuation marks are excluded from being identified as snippet boundaries. For example, the trailing period in the abbreviation "Dr." for a doctor can be excluded (e.g., "dr.XX"). An example of a regular expression syntax that helps to avoid identifying certain punctuation marks as sentence boundaries can be found, for example, in Section 3.2.2 of Rokach L. et al., Information Retrieval Journal, 11(6):499-538 (2008), the contents of which are incorporated herein by reference in whole for all purposes.

[0079] In some embodiments, machine learning models are used to split the raw text into sentences. Various methods for sentence formation are employed using known NLP libraries, including Google's SyntaxNet, Stanford's CoreNLP, NLTK's Phyton library, and spaCy, as described in Haris, MS et al., Journal of Information Technology and Computer Science, 5(3):279-92, which is incorporated herein by reference in its entirety for all purposes.

[0080] Traditional systems pass snippets to a pre-trained model (e.g., BERT (Transformer-based Bidirectional Encoded Representation)) and then aggregate them via snippet-level attention (described below). These systems retain only snippets that contain hits of any regular representation. Traditional approaches have several drawbacks. Firstly, because traditional systems define snippets using arbitrary frames around hits of regular representations, these techniques lose potentially important context that the model learns. Secondly, considering that these traditional systems only pay attention to snippets that mention any of the regular representations, other important insights within the notes are lost. In contrast, the extractor described herein identifies entire episodes that may have hits of a single regular representation. That is, little information is dropped or excluded, and additional information or context allows the model to learn to identify which snippets are most important. This improved methodology is likely to improve the generalizability of the model.

[0081] Referring again to Figure 2, the encoder 208 (e.g., a pre-trained model such as BERT) encodes the candidate 206 into a snippet representation 210, which is input to the aggregater 212 to obtain an episode representation 214. The episode representation 214 is then input to a linear component 216 that calculates a score (e.g., a value between 0 and 1, with a higher score indicating more matches). A threshold 220 is applied to this score 218 to obtain a decision 222. Each decision corresponds to a given episode and indicates whether the episode represents an instance of a clinical condition. The encoder 208, aggregater 212, linear component 216, and threshold 220 may be collectively referred to as a classifier 224, which may be run using the classifier module 50. In some embodiments, the linear component or model is an affine transformation of the episode representation 214. This transformation converts its embedding output into a single number that can be thresholded to a decision between {0,1}. These components are typically used as the final layer in modern neural network classifiers.

[0082] In some embodiments, the encoder 208 is a pre-trained BERT model (with pre-trained weights) that outputs a (contextualized) vector for each snippet. In some embodiments, the encoder 208 processes each snippet of a single episode and outputs a vector representation for each token within each snippet.

[0083] In some embodiments, the aggregator 212 uses attention to aggregate vectors for a single episode to obtain an episode representation 214. In some embodiments, an intra-attention mechanism aggregates each token (for a given snippet) into a single vector representation for that snippet. In some embodiments, an inter-attention mechanism aggregates the vector representations of each snippet from the intra-attention mechanism into a single vector representation for the entire episode. For illustrative purposes, the attention mechanisms described herein use vanilla attention rather than self-attention. Any method of aggregating multiple vectors together into a trainable or learnable parameter may be used. For example, a single vector sum may be used. In general, learnable aggregation can be performed by some embodiments using attention or any method of aggregating multiple vectors into a single vector.

[0084] Attention is a learned weighted sum of a set of inputs, which can be of arbitrary size. A machine learning pipeline may include a 3D tensor shape of (N, sequence_length, dim_size) at some point, where for each data point, there exists a set of vectors of length dim_size, each of which is sequence_length. These vectors can range from token embeddings to hidden states following a recurrent neural network (RNN). The ordering of these vectors is not important, but this information can be embedded via positional embeddings. The purpose of attention is to encode the original (N, sequence_length, dim_size) shape input into a weighted sum following sequence_length, and then reduce it to (N, dim_size) where each data point is represented by a single vector. This output can be useful as input to another layer or directly to a logistic head.

[0085] An attention mechanism is a learned weighted sum of a set of inputs. Instead of simply taking a sum, the attention layers are trained to focus attention on specific inputs when generating this sum. The most important inputs are given more weight. This is done across multiple attention heads—concurrent attention layers reading across the same inputs—and then aggregated into a final summary. A single attention head can be thought of as a search system with a set of keys, queries, and values. The attention mechanism learns to map queries (Q) to a set of keys (K) to find the most relevant input values ​​(V). The attention mechanism achieves this by computing a weighted sum where each input is weighted proportionally to its perceived importance (i.e., attention weight). This weighting is performed across all attention heads and then further downstream, summarized into a single weighted representation.

[0086] In some embodiments, the attention mechanism is a multi-head attention mechanism. That is, in some embodiments, each snippet or its encoded representation is input to a different attention head. Having multiple heads allows the attention mechanism to have more flexibility in attempting to aggregate information. Each individual head may direct attention to a different mode when aggregating, and the heads as a whole must converge to their underlying dispersion. In other words, multiple heads help allow the model to direct attention to various concepts.

[0087] Figure 5A is a schematic diagram of an exemplary mechanism 500 in some embodiments. In step (1), the mechanism takes as input a three-dimensional (3D) tensor shape (batch_size, max_seq_length, dim_model) representing the input as a set of embeddings whose order is not important. In most scenarios, sequences of the same size are not required across all data points. Often, padding is used to fit these dimensions (and therefore the descriptor of the maximum sequence length with respect to this dimension). In step (2), for each data point, an attention value is calculated by taking the dot product between the query and the set of keys. The final output is a set of attention weights for each attention head. A weighted sum is then calculated. In step (3), each of the input sequences following max_seq_length is then reduced to a single representation via a sum of embeddings weighted by the attention weights. This is done for each attention head. In step (4), finally, the attention head is reduced to a single representation via a weighted sum. In step (5), the final output is the shape (batch_size, dim_model) of a two-dimensional (2D) tensor representing a single dense representation for each data point.

[0088] Figure 5B shows an exemplary architecture 502 having a snippet encoder according to one embodiment. Figure 5C shows an exemplary architecture 504 having a concept encoder according to one embodiment. In one embodiment, the encoder 208 includes a snippet encoder and a concept encoder. The snippet encoder takes a set of snippet tokens for each episode and generates a single embedding for each episode. The concept encoder takes a set of concept tokens for each episode and generates a single embedding for each episode.

[0089] Referring to Figure 5B, as shown by Display (1), the snippet encoder expects the shape of a three-dimensional (3D) tensor (batch_size, max_snippet_len, max_num_snippets) as input. Each episode contains a set of snippets of a number equal to max_num_snippets, each of which contains a number of tokens equal to max_snippet_len for each snippet. The value itself is a token identifier that maps to a vocabulary of token embeddings. Note that for a given episode, the probability of finding the same number of snippets, and certainly snippets of exactly the same length, is very low (except in the case where exactly the same number of tokens are extracted for each snippet). Therefore, it is reasonable to assume some padding to fit these dimensions. During model training, padded tokens will be ignored.

[0090] In step (2), the 3D tensor is flattened to the shape of a two-dimensional (2D) tensor (batch_size * max_num_snippets, max_snippet_length), which is then fed through a snippet encoder. The task of the snippet encoder is to transform each token in the sequence into a learned representation. The information within the snippets is useful in this encoding task, but each snippet should be processed independently, and therefore the first dimension is reduced to a block of snippets of size max_num_snippets per episode. Another motivation for this transformation is practical: the snippet encoder (usually a pre-trained Transformer) expects a 2D tensor, otherwise it will produce an error.

[0091] In step (3), the flattened tensor is fed to a snippet encoder 506, which may be a Transformer-based encoder architecture such as BERT. The output of this encoder is the shape of the 3D tensor (batch_size * max_num_snippets, max_snippet_length, dim_model), which is the final hidden state of the model, where dim_model is the length of the dense representation generated by the encoder. This can be thought of as the set of embeddings generated by the model.

[0092] In step (4), the goal is to extract this 3D (or 4D if the first dimension is unpacked) into a single embedding (i.e., a 2D tensor) for each episode. The first pass to summarize this object is by token-level attention. The attention mechanism is a learned summary of the set for the input. Intrasnippet attention summarizes max_snippet_length, i.e., the set of embeddings per snippet, into a single vector. After passing through this layer, the output is (batch_size * max_num_snippets, dim_model).

[0093] In step (5), after obtaining the shape of this 2D tensor (batch_size*max_num_snippets, dim_model), some embodiments re-extract the dimension of max_num_snippets. This layer brings the dimension of max_num_snippets back into existence so that the output is (batch_size, max_num_snippets, dim_model). This tensor has a number of embeddings, each episode having max_num_snippets of length dim_model.

[0094] In step (6), the architecture performs intersnippet attention using the same attention mechanism (the same one used for token attention). The dimension of max_num_snippets is reduced to a single representation. After passing through this layer, the final output is (batch_size, dim_model), which is a single embedding per episode.

[0095] Next, referring to Figure 5C, as shown by Display (1), the concept encoder anticipates the shape of a 2D tensor (batch_size, max_concept_length) as input. Each episode contains a set of a number of concepts, max_num_snippets. The values ​​in this tensor are token identifiers that map to the vocabulary of concept embeddings. Note that it is highly unlikely that each episode contains the same number of concepts. Therefore, padding exists, just as in the case of snippet inputs. The concept encoder 502 includes an embedding layer. In step (2), the 2D set of concept identifiers is then passed to the embedding layer, which acts as a lookup table for concept embeddings. The output of this layer is the shape of a 3D tensor (batch_size, max_concept_length, dim_model). In step (3), the goal is to learn a weighted sum of concepts in a single representation per episode for concept-level attention, similar to snippet attention. This layer transforms (batch_size, max_concept_length, dim_model) into the final output shape (batch_size, dim_model).

[0096] Figure 6 shows a schematic diagram of an exemplary training flow 600 according to one embodiment. The training flow may be used to train the classifier 224 described above with reference to Figure 2. The training flow may be carried out using the training module 58. Alternatively, or additionally, another computer system may be used for training, in which case only the parameters of the classifier 50 may be retrieved and stored in memory 92 and / or 90.

[0097] In some embodiments, the training dataset 602 and the validation dataset 604 each contain episodes about patients. These datasets include episodes about patients with no clinical status (AFib in this embodiment) as shown in box 610, and episodes about patients with clinical status as shown in box 612. Each episode may correspond to a negative 614, positive 616, or indeterminate 618 indication regarding clinical status. Extraction step 606 extracts candidate episodes 620 from the training dataset 602 (e.g., using extractor 204). Extraction step 608 extracts candidate episodes 622 from the validation dataset 604 (e.g., using extractor 204) and rejects episode 624. The extracted candidate episodes 620 are used to train (626) a model (e.g., classifier 224). The scoring model 628 is used to score the model being trained. Using candidate 622 and rejected 624, a threshold 630 (e.g., maximum sensitivity at 90% PPV) is selected to obtain the trained model 632. The trained model 632 is also evaluated using candidate 622 and rejected 624 (634) (e.g., evaluated for sensitivity at 90% PPV).

[0098] Figure 7 shows an example of labels 700 for a dataset used to train a classifier, according to one or more embodiments. Each patient 702 is associated with a corresponding set of episodes, each episode corresponding to a time interval (time is shown on axis 704). Each episode is labeled (or identified) as a positive episode 708, a negative episode 710, or an unlabelable episode 706. A phenotype index day 712, for example, the first day a phenotype was identified in the patient (e.g., with a structured phenotype; in other words, the first manifestation of the clinical condition), may also be identified during the labeling process.

[0099] In some embodiments, the “positive” label is assigned only if the episode coincides with the first expression of the clinical condition in the EHR or EMR (e.g., determined by the structured phenotype). This is because later expressions (determined by the structured phenotype) are often simply picked up from some clinical history, while later episodes are usually not recorded in the notes because they relate to irrelevant issues. Similarly, in some embodiments, the “negative” label is assigned only to EHRs and EMRs of patients who have never been identified as having the clinical condition (e.g., from the structured phenotype).

[0100] In some embodiments, the training module 58 performs the following steps to train the classifier 50. The training module 58 may cause the input / output module 64, the language pattern recognition module 40, the partitioning module 44, and / or the clustering module 56 to perform one or more of these steps to train the classifier 50. The input / output module 64 acquires multiple episode records (e.g., records in the training dataset 62) in electronic form. Each episode record in the multiple episode records (i) contains corresponding unstructured clinical data from electronic medical records (EMRs) or electronic health records (EHRs) relating to individual patients of multiple patients, and (ii) is associated with a corresponding date range. The training module 58 assigns a corresponding label 60 to each episode record in a plurality of episode records, determining whether the corresponding structured data in the EMR or EHR contains a medical code that is (i) associated with a clinical condition and (ii) associated with a corresponding date range, thereby identifying a first plurality of episode records that have an assigned label indicating (i) a positive clinical condition, and a second plurality of episode records that have an assigned label indicating a negative clinical condition. The splitting module 44 splits the corresponding unstructured clinical data for each episode record in the first plurality of episode records and the second plurality of episode records into a plurality of corresponding snippets, where each snippet has approximately the same number of tokens. The training module 58 inputs, for each of the episode records in the first quasi-multiple episode records and the second quasi-multiple episode records, a plurality of corresponding snippets for each individual episode record, via an attention mechanism, into an untrained or partially trained model (e.g., aggregation module 52), which independently applies the corresponding weights to each snippet within the plurality of corresponding snippets. The untrained or partially trained model includes a plurality of parameters that are learned during training.These parameters are used to obtain corresponding predictions about whether individual episode records represent instances of clinical conditions that are output by the model. The training module 58 updates all or a subset of the parameters for each episode record in the first set of quasi-episode records and the second set of quasi-episode records, using (i) a comparison with the corresponding prediction output by the model and (ii) the corresponding label, thereby training the model to identify episode records that represent instances of clinical conditions.

[0101] In some embodiments, the training module 58 further identifies a third set of episodic records that have been assigned labels indicating that the clinical condition is indeterminate. For example, some phenotypes (e.g., cases of complex stroke) may not be identifiable from the clinical record. For example, the treating physician may not have clearly stated the final diagnosis in the notes. In some embodiments, such records are labeled as indeterminate, rather than positive or negative.

[0102] In some embodiments, the training module 58 performs the following operations for each episode record in a plurality of episode records: (a) assign a corresponding label indicating a positive clinical condition when the corresponding EMR or EHR includes a medical code that is associated with a clinical condition and (ii) with a corresponding date range; (b) assign a corresponding label indicating a negative clinical condition when the corresponding EMR or EHR does not include a medical code that is associated with a clinical condition and (ii) with an arbitrary date range; and (c) assign a corresponding label indicating an uncertain clinical condition when the corresponding EMR or EHR includes a medical code that is associated with an individual date range that is after the corresponding date range.

[0103] In some embodiments, the training module 58 performs the following operation for each episode record: assigns a corresponding label that is uncertain with respect to the clinical condition when the corresponding EMR or EHR includes a medical code which is associated with (i) a clinical condition and (ii) a range of individual dates preceding the corresponding range of dates.

[0104] The extractor-classifier networks described herein may be used as phenotyping models for identifying patients with disease and / or for identifying other inclusion or exclusion criteria in a population health platform. Furthermore, the extractor-classifier networks may be used to identify or define other-purpose clinical entities, such as drugs, treatments, or devices, in other commercial applications such as data structuring and phenotyping as a service for generating disease cohorts. For a new hospital system, the techniques described herein can be used to identify a list of patients to exclude and / or a list of patients with specific clinical conditions to be displayed in the initial patient funnel. The techniques may also be used to determine a new diagnosis with respect to a clinical condition by comparing the output with previous results. The patient funnel can be visualized, and the model output may be linked to subsequent diagnoses. Using the patient funnel, all episodes identified as disease-diagnosing episodes can be compared with the output of a previous risk prediction operation for a given episode. In this way, it is possible to check whether the risk prediction is high for episodes that were ultimately diagnosed with the disease.

[0105] In some embodiments, phenotypic models may be used in the deployment of medical devices in the field. These models may be applied as inclusion or exclusion criteria for patient cohort selection. Healthcare systems, healthcare providers, researchers, and third parties, including pharmaceutical and medical technology companies, require phenotypes to conduct clinical analyses. Those with access to clinical notes may use the techniques described herein to generate more accurate phenotypes or to define their various patient cohorts or outcomes. These techniques may be used in filtering or identifying early patients for future research, such as any population health management tools, predictive algorithms, retrospective studies, or clinical trials, or to improve services provided by electronic health records.

[0106] In some embodiments, the models described herein identify whether a given chunk of text contains a disease-positive contribution to the patient. Standard positive examples include positive references to a clinical condition such as "Patient is diagnosed with <clinical condition> on ECG," "Patient is currently presenting with a clinical condition," "Patient has been diagnosed with a clinical condition in the past," or "Patient has a history of a clinical condition." Standard negative examples include no reference to a clinical condition and non-primary references to a clinical condition (e.g., "Patient is at risk of developing a clinical condition," or "Patient has a family history of a clinical condition"). Using atrial fibrillation (AFib) as an example, negative examples may include "Patient was suspected of having AFib but presents with normal sinus rhythm," or "No AFib, or atrial flutter was observed."

[0107] Figures 8A–8G show flowcharts of an exemplary method 800 for phenotyping clinical data according to some embodiments. The method is carried out by a module of a computer system 100, as detailed below.

[0108] In some embodiments, referring to block 802, the input / output module 64 acquires multiple episode records in electronic form. Each individual episode record within the multiple episode records contains corresponding unstructured clinical data from the electronic medical record (EMR) or electronic health record (EHR) for each individual patient of multiple patients. Generally, the EMR or EHR contains both structured data (such as invoice creation codes) and unstructured data (such as clinical notes). The input / output module 64 can select only the unstructured data from the patient's EMR or EHR.

[0109] In some embodiments, referring to block 804, for each episode record within a plurality of episode records, the input / output module 64 retrieves corresponding unstructured clinical data from a plurality of medical assessments stored in the EMR or EHR for the individual patient.

[0110] In some embodiments, referring to block 806, the clustering module 56 selects multiple medical assessments by clustering all or some of the medical assessments stored in the EMR or EHR for individual patients to obtain one or more corresponding medical assessment clusters, and by aggregating unstructured clinical data corresponding to each of the individual medical assessments within each of the one or more corresponding medical assessment clusters to form individual episode records.

[0111] In some embodiments, referring to block 808, the clustering is at least partially time-based clustering (e.g., clustering based on the date of medical assessments stored in the EMR or EHR).

[0112] In some embodiments, referring to block 810, the clustering is one-dimensional clustering. Various clustering methods may be used, such as kernel density estimation (KDE), sliding window, and machine learning.

[0113] In some embodiments, referring to block 812, for each episode record within a plurality of episode records, the input / output module 64 retrieves the corresponding unstructured clinical data from a single medical assessment stored in the EMR or EHR.

[0114] In some embodiments, referring to block 814, each episode record within multiple episode records does not include a corresponding structured clinical record from the EMR or EHR. Some embodiments do not include structured data. Some embodiments include such data depending on the application (e.g., the application requires analysis of specific structured data such as invoice generation codes). Models using only notes or models using unstructured data are better generalized than models using only structured data.

[0115] In some embodiments, referring to block 814, the language pattern recognition module 40 filters multiple episode records by language pattern recognition to identify quasi-episode records, each containing an expression related to a clinical state in the corresponding unstructured clinical data.

[0116] In some embodiments, referring to block 816, language pattern recognition includes matching one or more regular expressions against corresponding unstructured clinical data for individual episode records within multiple episode records, thereby identifying quasi-multiple episode records. Examples of regular expressions are described above with reference to Figure 2. Further examples are available at developer.mozilla.org / en-US / docs / Web / JavaScript / Guide / Regular_Expressions / Cheatsheet, which is incorporated herein by reference.

[0117] In some embodiments, referring to block 818, language pattern recognition includes a machine learning model trained to identify language associated with a clinical condition. In some embodiments, the trained machine learning model has high recall and can reduce the input set of episodic records to a universe of candidates having a higher prevalence than the input set.

[0118] Referring to block 820, in some embodiments, the clinical condition is atrial fibrillation. The techniques described herein may be used to phenotype any clinical disease, condition, or clinical state (e.g., the presence of a device-like ICD / pacemaker, the occurrence of a procedure or examination, any diagnosis, drug). The natural language processing techniques described herein may be used to phenotype heart failure, stroke, transient ischemic attack, and myocardial infarction (heart attack).

[0119] Referring to block 822, in some embodiments, the splitting module 44 splits the corresponding unstructured clinical data into corresponding snippets for each individual episode record within a quasi-multiple episode record. Each individual snippet within the corresponding snippets contains a corresponding set of one or more tokens.

[0120] In some embodiments, referring to block 824, the corresponding unstructured clinical data splitting is performed before filtering of multiple episode records.

[0121] In some embodiments, referring to block 826, the splitting module 44 splits the corresponding unstructured clinical data after filtering multiple episode recordings.

[0122] In some embodiments, referring to block 828, each snippet within the corresponding snippet has approximately the same number of tokens.

[0123] In some embodiments, referring to block 830, for each individual episode record within a quasi-multiple episode record, each individual snippet within a corresponding snippet has a corresponding number of tokens, and this corresponding number of tokens is within 25% of the corresponding number of tokens for each of the other individual snippets within the corresponding snippet. In some embodiments, snippets may be of various sizes but may be padded to a certain set size (e.g., 512 snippets × 256 tokens per snippet). This size may be determined based on computational constraints (e.g., the larger the amount of computational resources, the larger the snippet size and / or number of snippets). In some embodiments, each token is aggregated using intraattention so that no distribution is required for the tokens.

[0124] In some embodiments, referring to block 832, for individual episode records within a quasi-multiple episode records, the splitting module 44 splits the corresponding unstructured clinical data by (i) tokenizing the corresponding unstructured clinical data to obtain multiple tokens, and (ii) segmenting the multiple tokens to obtain multiple segments. Each individual segment within the multiple segments has approximately the same number of tokens, (iii) ranking the individual segments within the multiple segments based on the value of the tokens within each individual segment, and (iv) removing one or more individual segments from the multiple segments based on the ranking, thereby generating a corresponding multiple snippet with respect to the episode record.

[0125] In some embodiments, referring to block 834, for each episode record within a quasi-multiple episode record, the splitting module 44 splits the corresponding unstructured clinical data by (i) segmenting the corresponding unstructured clinical data to obtain multiple segments, each of which individual segments within the multiple segments contains individual portions of the corresponding unstructured clinical data; (ii) tokenizing the individual portions of the corresponding unstructured clinical data within each individual segment within the multiple segments to obtain multiple tokenized segments; (iii) splitting individual tokenized segments within the multiple tokenized segments that have a corresponding number of tokens exceeding a threshold for the number of tokens to obtain a second set of multiple tokenized segments; (iv) ranking the individual segments within the second set of multiple tokenized segments based on the token values ​​within each of the individual tokenized segments; and (v) removing one or more individual tokenized segments from the second set of multiple tokenized segments based on the ranking to generate a corresponding set of multiple snippets for each individual episode record.

[0126] In some embodiments, referring to block 836, for each episode record in a quasi-multiple episode record, the splitting module 44 splits the corresponding unstructured clinical data by (i) segmenting the corresponding unstructured clinical data sentence by sentence to obtain multiple segments, each of which individual segments in the multiple segments contains an individual portion of the corresponding unstructured clinical data; (ii) tokenizing the individual portion of the corresponding unstructured clinical data in each of the individual segments in the multiple segments to obtain multiple tokenized segments; (iii) splitting the multiple tokenized segments that have a corresponding number of tokens exceeding a first threshold for the number of tokens to obtain a second multiple tokenized segment; (iv) merging the second multiple tokenized segments that have a corresponding number of tokens below a second threshold for the number of tokens to obtain a third multiple tokenized segment; (v) ranking the individual segments in the third multiple tokenized segment based on the token values ​​in each of the individual tokenized segments; and (vi) removing one or more individual tokenized segments from the third multiple tokenized segment based on the ranking to generate a corresponding multiple snippet for each individual episode record.

[0127] In some embodiments, referring to block 838, the ranking is based at least in part on a scoring system that rewards the presence of tokens found in a priority list of tokens. Terms that may be present in the priority list include, among others, terms found in the Unified Medical Terminology System (UMLS) metathesaurus. Examples include cardiac, discharge summary, cardiology, apixaban, metoprolol, aspirin, physical examination, atrium, and heart failure.

[0128] In some embodiments, referring to block 840, the scoring system penalizes the presence of tokes found in a low-priority list of tokens. In some embodiments, snippets containing preferred snippets are moved to the top of the priority list (without using a separate low-priority list). For example, if the number of snippets exceeds a pre-specified maximum number of snippets (which is a rare occurrence), in some embodiments, the lowest-priority snippets are truncated. In some embodiments, terms related to patient advice categories (e.g., "don't smoke") or location-specific boilerplate language within notes are given lower priority. In some embodiments, notes are data-mined to obtain user input about the top M snippets used across many different patients. These snippets may be boilerplate and not very useful information. In some situations, there are automated or templated notes about patients who do not keep appointments or patients who have received reminder calls. There are more administrative type notes and case management notes that may be given lower priority. In some embodiments, notes are given lower priority based on their type.

[0129] In some embodiments, referring to block 842, the corresponding snippets are a predetermined number of snippets.

[0130] Exemplary operation of the split module 44 is further described above with reference to Figures 4A, 4B, and 4C in some embodiments.

[0131] In some embodiments, referring to block 842, the classifier 50 predicts, for each episode record in a quasi-multiple episode record, whether the individual episode record represents an instance of a clinical state, based on a number of corresponding snippets for the individual episode record. The classifier 50 includes a first part (aggregation module 52) and a second part (interpretation module 54). The first part includes an aggregation function that aggregates the number of corresponding snippets to output a corresponding representation for the individual episode record. The second part interprets the corresponding representation to output a corresponding prediction about whether the individual episode record represents an instance of a clinical state.

[0132] In some embodiments, referring to block 844, the first part of the classifier 50 includes a multi-head encoder that outputs a corresponding contextualized token tensor for each individual token in a corresponding set of one or more tokens for each individual snippet in a plurality of corresponding snippets relating to each individual episode record in each individual episode record in a quasi-multiple episode record, thereby forming a plurality of corresponding contextualized token tensors relating to the individual snippet.

[0133] In some embodiments, referring to block 846, the first part of the classifier 50 further includes a multi-head intra-attention mechanism that, for each individual episode record in a quasi-multiple episode record, aggregates a plurality of corresponding contextualized token tensors corresponding to each individual snippet in a plurality of corresponding snippets, outputs a corresponding contextualized snippet tensor, thereby forming a plurality of corresponding contextualized snippet tensors corresponding to each individual episode record.

[0134] In some embodiments, referring to block 848, the first part of the classifier 50 further includes an inter-attention mechanism that aggregates a number of corresponding contextualized snippet tensors for each individual episode record in a quasi-multiple episode record to output a corresponding contextualized episode record tensor for each individual episode record.

[0135] In some embodiments, referring to block 850, the second part of the classifier 50 includes a model that, for each individual episode record within a quasi-multiple episode records, outputs a corresponding prediction about whether the individual episode record represents an instance of a clinical condition, in response to a corresponding representation being input to the model for the individual episode record.

[0136] In some embodiments, referring to block 852, the second part of the classifier 50 includes a model selected from the group consisting of neural networks, support vector machines, naive Bayes algorithms, nearest neighbor algorithms, boost tree algorithms, random forest algorithms, convolutional neural networks, decision trees, regression algorithms, and clustering algorithms.

[0137] In some embodiments, referring to block 854, the second part of the classifier 50 includes a linear transformation that, for each episode record in a quasi-multiple episode records, converts the individual outputs of the first part of the classifier into corresponding scalar numbers compared to a threshold to output the corresponding prediction.

[0138] In some embodiments, referring to block 856, the linear transformation is an affine transformation.

[0139] In some embodiments, referring to block 858, the classifier 50 includes at least 500 parameters, at least 1,000 parameters, at least 5,000 parameters, at least 10,000 parameters, at least 50,000 parameters, at least 100,000 parameters, at least 250,000 parameters, at least 500,000 parameters, at least 1,000,000 parameters, at least 10M parameters, at least 100M parameters, at least 1MM parameters, at least 10MM parameters, or at least 100MM parameters.

[0140] According to some embodiments, exemplary operation of the classifier 50 is further described above with reference to Figures 5A, 5B, and 5C.

[0141] In some embodiments, referring to block 860, the input / output module 64 labels each individual episode record in a quasi-multiple episode records that is predicted to represent an instance of a clinical condition, forming a set of episode records, where each individual episode record in the set of episode records represents an instance of a clinical condition.

[0142] In some embodiments, referring to block 862, the input / output module 64 trains a model to predict clinical condition outcomes using a set of episode recordings. [Examples]

[0143] Unstructured clinical notes from EHR records, labeled for atrial fibrillation as, for example, positive (reflecting an atrial fibrillation episode) or negative (reflecting a non-atrial fibrillation episode), were collected from local healthcare systems and divided into a training set of approximately 29 million code-labeled episodes and a holdout set of approximately 1.8 million code-labeled episodes. The training set was used to train a classifier with a pre-trained encoder (BERT, as described in Devlin J. et al., arXiv:1810.04805), a multi-head intra-snippet attention mechanism, an aggregate inter-snippet attention mechanism, and a linear transformation, as illustrated in Figure 2, for example.

[0144] Model performance was calculated using codebase labels against the holdout set, with unextracted episodes scored as zero. Target identity-anonymized chart reviews were also performed for discrepancies between NLP model outputs and codebase labels. Figure 9 shows the area under the precision-recall curve (AUPRC) for a validation set of 1.8 million holdout episodes in one embodiment. The NLP model achieved an AUPRC of 0.91. After thresholding, the NLP model achieved 87% recall and 89% precision. An identity-anonymized review of selected episodes showed that the NLP model was accurate with 90% discrepancy, where the codebase approach incorrectly labeled them as negative.

[0145] Figure 10 shows the results of an interpretable model for snippets of hypothetical text, according to one embodiment. The results demonstrate the ability to distinguish between true positive and non-major mentions of atrial fibrillation. Snippets with green outlines are labeled positive, while those with red outlines are labeled negative. The heatmap behind each word represents the model's attention weight, with higher weights indicating a correlation with words the model found more important during classification.

[0146] In this way, NLP models can be used to automatically label the presence or absence of clinical conditions such as atrial fibrillation within clinical notes. The systems and methods described herein can offer higher accuracy and generalizability compared to code-based labeling methods.

[0147] conclusion The above explanation has been based on specific implementations for illustrative purposes. However, the illustrative considerations above are not intended to be exhaustive or to limit implementations to the explicit forms disclosed. Many modifications and variations are possible considering the above teachings. Implementations have been selected and described to best illustrate the principles and practical applications. This will enable those skilled in the art to make the most of this implementation and various variations suited to specific anticipated applications.

Claims

1. A method for phenotyping clinical data, To acquire multiple episode records in electronic format, wherein each of the multiple episode records contains corresponding unstructured clinical data from the electronic medical records (EMRs) or electronic health records (EHRs) of individual patients of multiple patients, The process involves filtering the multiple episode records using language pattern recognition to identify a quasi-multiple episode records, each containing an expression related to the clinical state within the corresponding unstructured clinical data, For each of the episode records within the aforementioned quasi-multiple episode records, the corresponding unstructured clinical data is divided into corresponding snippets, each of which snippets contains a corresponding set of one or more tokens; A method comprising predicting, for each of the episode records in the quasi-multiple episode records, whether an individual episode record represents an instance of the clinical condition, by inputting the corresponding multiple snippets for the individual episode record into a classifier comprising a first part and a second part, wherein the first part comprises an aggregate function that aggregates the corresponding multiple snippets and outputs a corresponding expression for the individual episode record, and the second part interprets the corresponding expression and outputs a corresponding prediction as to whether the individual episode record represents an instance of the clinical condition.

2. The method according to claim 1, wherein, for each episode record within the plurality of episode records, the corresponding unstructured clinical data is obtained from a plurality of medical evaluations stored in the EMR or EHR concerning the individual patient.

3. The method according to claim 2, wherein the plurality of medical evaluations are selected by (i) clustering all or some of the medical evaluations stored in the EMR or EHR for each individual patient to obtain one or more corresponding medical evaluation clusters, and (ii) aggregating unstructured clinical data corresponding to each of the medical evaluations within each of the one or more corresponding medical evaluation clusters to form the individual episode records.

4. The method according to claim 3, wherein the clustering is at least partially time-based clustering.

5. The method according to claim 3 or 4, wherein the clustering is one-dimensional clustering.

6. The method according to any one of claims 1 to 5, wherein for each episode record in the plurality of episode records, the corresponding unstructured clinical data is obtained from a single medical assessment stored in the EMR or EHR.

7. The method according to any one of claims 1 to 6, wherein each of the episode records in the plurality of episode records does not include corresponding structured clinical data from the EMR or EHR.

8. The method according to any one of claims 1 to 7, wherein the language pattern recognition includes matching one or more regular expressions against the corresponding unstructured clinical data for each of the episode records in the plurality of episode records, thereby identifying a quasi-multiple episode records.

9. The method according to any one of claims 1 to 7, wherein the language pattern recognition comprises a machine learning model trained to identify language related to the clinical condition.

10. The method according to any one of claims 1 to 9, wherein the clinical condition is atrial fibrillation.

11. The method according to any one of claims 1 to 10, wherein the division of the corresponding unstructured clinical data is performed before filtering the plurality of episode records.

12. The method according to any one of claims 1 to 10, wherein the division of the corresponding unstructured clinical data is performed after filtering of the plurality of episode records.

13. The method according to any one of claims 1 to 12, wherein each snippet in the corresponding plurality of snippets has approximately the same number of tokens.

14. The method according to claim 13, wherein for each episode record in the quasi-multiple episode records, each snippet in the corresponding plurality of snippets has a corresponding number of tokens which is within 25% of the corresponding number of tokens for each of the other individual snippets in the corresponding plurality of snippets.

15. For each episode record within the aforementioned quasi-multiple episode records, the corresponding unstructured clinical data is divided as follows: The process involves tokenizing the corresponding unstructured clinical data and obtaining multiple tokens, The process involves segmenting the aforementioned multiple tokens to obtain multiple segments, wherein each segment within the multiple segments has approximately the same number of tokens. Ranking individual segments within the multiple segments based on the token values ​​within each segment, The method according to any one of claims 1 to 13, comprising removing one or more individual segments from the plurality of segments based on the ranking, thereby generating the corresponding plurality of snippets with respect to the individual episode recordings.

16. For each of the individual episode records within the aforementioned quasi-multiple episode records, the corresponding unstructured clinical data is divided as follows: The process involves segmenting the corresponding unstructured clinical data to obtain multiple segments, wherein each segment within the multiple segments includes individual parts of the corresponding unstructured clinical data. In each of the aforementioned multiple segments, the individual parts of the corresponding unstructured clinical data are tokenized to obtain multiple tokenized segments. In the aforementioned plurality of tokenized segments, individual tokenized segments having a corresponding number of tokens exceeding a threshold for the number of tokens are divided to obtain a second plurality of tokenized segments. Ranking individual segments within the second plurality of tokenized segments based on the token values ​​within each of the tokenized segments, The method according to any one of claims 1 to 13, further comprising removing one or more individual tokenized segments from the second plurality of tokenized segments based on the ranking, thereby generating the corresponding plurality of snippets with respect to the individual episode record.

17. For each of the individual episode records within the aforementioned quasi-multiple episode records, the corresponding unstructured clinical data is divided as follows: The method involves segmenting the corresponding unstructured clinical data sentence by sentence to obtain multiple segments, wherein each segment within the multiple segments contains individual parts of the corresponding unstructured clinical data. In each of the aforementioned multiple segments, the individual parts of the corresponding unstructured clinical data are tokenized to obtain multiple tokenized segments. In the aforementioned plurality of tokenized segments, the individual tokenized segments having a corresponding number of tokens exceeding a first threshold for the number of tokens are divided to obtain a second plurality of tokenized segments. In the aforementioned second set of tokenized segments, the individual tokenized segments having a corresponding number of tokens that falls below a second threshold for the number of tokens are merged to obtain a third set of tokenized segments. Ranking individual segments within the third set of tokenized segments based on the token values ​​within each of the tokenized segments, The method according to any one of claims 1 to 13, comprising removing one or more individual tokenized segments from the third plurality of tokenized segments based on the ranking, thereby generating the corresponding plurality of snippets with respect to the individual episode record.

18. The method according to any one of claims 15 to 17, wherein the ranking is at least in part based on a scoring system that rewards the presence of tokens found on a priority list of tokens.

19. The method according to claim 18, wherein the scoring system penalizes the presence of a token in a low-priority list.

20. The method according to any one of claims 1 to 19, wherein the corresponding plurality of snippets is a predetermined number of snippets.

21. The method according to any one of claims 1 to 20, wherein the first part of the classifier comprises a multi-head encoder that outputs a corresponding contextualized token tensor for each of the tokens in a corresponding set of one or more tokens for each of the snippets in a plurality of corresponding snippets relating to each of the episode records in the quasi-multiple episode records, thereby forming a plurality of corresponding contextualized token tensors relating to each individual snippet.

22. The method according to claim 21, wherein the first part of the classifier further comprises a multi-head intra-attention mechanism that aggregates a plurality of corresponding contextualized token tensors corresponding to each of the snippets in the plurality of corresponding snippets for each of the episode records in the quasi-multiple episode records, outputs a corresponding contextualized snippet tensor, thereby forming a plurality of corresponding contextualized snippet tensors corresponding to each individual episode record.

23. The method according to claim 22, wherein the first part of the classifier further comprises an inter-attention mechanism for each of the episode records in the quasi-multiple episode records, aggregating a plurality of corresponding contextualized snippet tensors to output a corresponding contextualized episode record tensor for each individual episode record.

24. The method according to claim 23, wherein the second portion of the classifier comprises a model that, for each of the episode records in the quasi-multiple episode records, outputs a corresponding prediction as to whether the individual episode record represents an instance of a clinical condition, in response to the input of an episode record tensor contextualized with respect to the individual episode record.

25. The method according to any one of claims 1 to 24, wherein the second part of the classifier comprises a model selected from the group consisting of neural networks, support vector machines, naive Bayes algorithms, nearest neighbor algorithms, boost tree algorithms, random forest algorithms, convolutional neural networks, decision trees, regression algorithms, and clustering algorithms.

26. The method according to any one of claims 1 to 24, wherein the second part of the classifier comprises a linear transformation that, for each episode record in the quasi-multiple episode records, converts each output of the first part of the classifier into a corresponding scalar number compared to a threshold and outputs a corresponding prediction.

27. The method according to claim 26, wherein the linear transformation is an affine transformation.

28. The method according to any one of claims 1 to 27, wherein the model includes at least 500 parameters, at least 1,000 parameters, at least 5,000 parameters, at least 10,000 parameters, at least 50,000 parameters, at least 100,000 parameters, at least 250,000 parameters, at least 500,000 parameters, at least 1,000,000 parameters, at least 10M parameters, at least 100M parameters, at least 1MM parameters, at least 10MM parameters, or at least 100MM parameters.

29. The method according to any one of claims 1 to 28, wherein, in the quasi-multiple episode records, each episode record that is expected to represent an instance of the clinical condition is labeled to form a set of episode records, and each episode record in the set of episode records represents an instance of the clinical condition.

30. The method according to claim 29, further comprising training a model to predict the outcome of the clinical condition using the set of episode recordings.

31. The method according to claim 30, further comprising inputting electronic health data relating to a test patient into the model and outputting a likely outcome relating to the clinical condition of the test patient.

32. The method according to any one of claims 1 to 31, further comprising stratifying each individual episode record within all or part of the quasi-multiple episode records into corresponding bins within a plurality of bins, based at least on the corresponding prediction of whether the individual episode record represents an instance of the clinical condition.

33. The method according to claim 32, wherein each of the multiple bottles represents an individual probability range among a plurality of probability ranges that each of the multiple patients has the clinical condition.

34. The method according to claim 32 or 33, wherein each of the multiple bottles represents an individual severity level among the multiple severity levels of the clinical condition.

35. The method according to claim 33 or 34, further comprising determining an individual disease state with respect to a patient having corresponding episode records stratified into individual bins within the plurality of bins.

36. The method according to any one of claims 33 to 35, further comprising providing individual recommendations for the treatment of the clinical condition with respect to a patient having a corresponding episode record stratified in individual bins within the plurality of bins.

37. The method according to any one of claims 33 to 36, further comprising administering individual treatments for the clinical condition to a patient having corresponding episode records stratified in individual bottles within the plurality of bottles.

38. A method for phenotyping clinical data, This involves obtaining episode records, including unstructured clinical data, in electronic format from the patient's electronic medical record (EMR) or electronic health record (EHR), and The unstructured clinical data is divided into multiple snippets, each of which contains a corresponding set of one or more tokens. A method for predicting whether an episode record represents an instance of a clinical condition by inputting the plurality of snippets into a classifier comprising a first part and a second part, wherein the first part includes an aggregate function that aggregates the plurality of snippets and outputs a representation relating to each episode record, and the second part interprets the representation and outputs a corresponding prediction as to whether the episode record represents an instance of the clinical condition.

39. The method of claim 38, further comprising determining a disease state with respect to the patient if the corresponding prediction indicates that the episode record represents an instance of the clinical state.

40. The method according to claim 38 or 39, further comprising providing a recommendation for the treatment of the clinical condition if the corresponding prediction indicates that the episode record represents an instance of the clinical condition.

41. The method according to any one of claims 38 to 40, further comprising administering treatment to the patient with respect to the clinical condition if the corresponding prediction indicates that the episode record represents an instance of the clinical condition.

42. A method for training a natural language processing model for phenotyping clinical data, Acquiring multiple episode records in electronic format, wherein each episode record within the multiple episode records (i) includes corresponding unstructured clinical data from individual patient electronic medical records (EMRs) or electronic health records (EHRs) of multiple patients, and (ii) is associated with a corresponding date range; For each episode record in the plurality of episode records, at least, assign a corresponding label to whether an individual episode record represents an instance of a clinical state by determining whether the corresponding structured data in the EMR or EHR includes a medical code that is (i) associated with a clinical state and (ii) associated with a corresponding date range, thereby identifying a first plurality of episode records that have been assigned a label indicating that they are (i) positive with respect to the clinical state, and a second plurality of episode records that have been assigned a label indicating that they are (ii) negative with respect to the clinical state. The division of the corresponding unstructured clinical data for each episode record within the first and second quasi-episode records is to divide the data into corresponding snippets, wherein each snippet within the corresponding snippets has approximately the same number of tokens. For each episode record in the first quasi-multiple episode records and the second quasi-multiple episode records, input the corresponding multiple snippets relating to the individual episode record into an untrained or partially trained model via an attention mechanism, which applies corresponding weights to each snippet independently within the multiple corresponding snippets, wherein the untrained or partially trained model includes multiple parameters, inputs the snippets, and obtains corresponding predictions as to whether each episode record represents an instance of the clinical state output from the model. A method comprising: for each of the episode records in the first quasi-episodic records and the second quasi-episodic records, updating all or a subset of a number of parameters by comparing (i) a corresponding predictive output from the model with (ii) a corresponding label, thereby training the model to identify episode records that represent instances of clinical conditions.

43. The method according to claim 42, wherein the assignment further identifies a third plurality of episode records having an assigned label that is indeterminate with respect to the clinical condition.

44. The aforementioned assignment applies to each individual episode record within the plurality of episode records. If the corresponding EMR or EHR includes a medical code that (i) is associated with the clinical condition and (ii) is associated with the corresponding date range, then assign a corresponding label that is positive with respect to the clinical condition, If the corresponding EMR or EHR does not include a medical code that (i) is associated with the clinical condition and (ii) is associated with any range of dates, then assign a corresponding label that indicates the clinical condition is negative. The method of claim 43, wherein the corresponding EMR or EHR includes a medical code which is associated with (i) the clinical condition and (ii) a range of individual dates which is after the corresponding range of dates, the method of assigning a corresponding label which is indeterminate with respect to the clinical condition.

45. The aforementioned assignment applies to each individual episode record, The method according to claim 44, wherein the corresponding EMR or EHR includes a medical code which is associated with (i) the clinical condition and (ii) a range of individual dates preceding the corresponding range of dates, the corresponding label which is indeterminate with respect to the clinical condition.

46. A computer system, One or more processors, A computer system comprising: a non-temporary computer-readable medium which, when executed by one or more processors, includes computer-executable instructions that cause the processors to carry out the method according to any one of claims 1 to 45.

47. A non-temporary computer-readable storage medium that stores program code instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 45.

Images