Dynamically adapting anticipatory interfaces for entering and using diagnostic data in electronic health records

Abstract

A method for dynamically adapting an anticipatory user interface of an electronic health record system involves: determining, by the electronic health record system and using Bayesian inference, for a patient, a differential diagnosis list that includes a subset of diagnoses selected from potential diagnoses, determining, by the electronic health record system, for the patient, an observation list that includes a subset of observations selected from potential observations, the subset of observations selected based on having a largest utility for disambiguation of the differential diagnosis list, and dynamically adapting the anticipatory user interface to operate in a mode that depends on the differential diagnosis list and the observation list. What is displayed to a physician and how an ambiguous input received from the physician is interpreted is mode-dependent, thereby resulting in different content being entered into an electronic health record of the patient, depending on the mode.

Description

Dynamically Adapting Anticipatory Interfaces for Entering and Using Diagnostic Data in Electronic Health Records BACKGROUND

[0001] When a valuable object such as a human body malfunctions, it is generally desirable to identify efficiently the cause of the malfunction by making those observations that are most likely to differentiate among the possible causes of the malfunction. In clinical medicine, the object is a patient, the cause is a diagnosis, and the observer is a clinician such as a physician or dentist. The human body is a very complex object subject to many different causes of malfunction and from which can be obtained a very large number of possible observations from procedures such as a medical history, a physical examination, and various laboratory tests and specialty consultations. It is desirable for the results of these observations and the diagnoses thereby concluded to be recorded and easily retrievable as electronic health records (EHRs), both to document the care of individual patients and to mine for insights into the probabilities that the current and future patients might have particular diagnoses. Perhaps the most common approach to recording the observations and resulting diagnoses is for the clinician to type or dictate a free text, narrative note after completing each clinical encounter. Studies have shown that such notes often contain omissions and errors. Furthermore, it is difficult to extract the information that they contain into the sorts of objective and coded data required for statistical analysis, even using modern natural language processors. An alternative is to require the clinician to enter their observations in an objective and coded format as they interact with their patient, but this tends to interfere with and distract from those interactions. As the range of possible observations increases to deal with a larger range of possible diagnoses, the clinician finds it increasingly difficult to navigate to the relevant part of the data entry system. A human clerk with detailed knowledge of the structure of the data entry system could follow instructions from the clinician during the clinical encounter but this would be expensive and disruptive.SUMMARY

[0002] In general, in one aspect, one or more embodiments relate to a method for dynamically adapting an anticipatory user interface of an electronic health record system, the method comprising: determining, by the electronic health record system and using Bayesian inference, for a patient, a first differential diagnosis list comprising a first subset of diagnoses selected from a plurality of potential diagnoses; determining, by the electronic health recordsystem, for the patient, a first observation list comprising a first subset of observations selected from a plurality of potential observations, the first subset of observations selected based on having a largest utility for disambiguation of the first differential diagnosis list; dynamically adapting the anticipatory user interface to operate in a first mode, based on the first differential diagnosis list and the first observation list, wherein the operating of the anticipatory user interface in the first mode comprises: displaying, in the anticipatory user interface, the first differential diagnosis list to a physician; displaying, in the anticipatory user interface, for each diagnosis of the first subset of diagnoses, a current probability that the diagnosis accounts for previously made observations upon the patient; displaying, in the anticipatory user interface, the first observation list to the physician; displaying, in the anticipatory user interface, for each observation on the first observation list, a current utility of the observation to distinguish among the first subset of diagnoses in the first differential diagnosis list; after displaying the first observation list to the physician, receiving, by the anticipatory user interface, a first ambiguous input from the physician; resolving the first ambiguous input by probabilistically determining an association of the first ambiguous input with at least one particular observation of the first observation list; and obtaining at least one value for the at least one particular observation and enter the at least one value into an electronic health record of the patient.

[0003] A non-transitory machine-readable medium comprising a plurality of machine-readable instructions executed by one or more processors associated with an electronic health record system, the plurality of machine-readable instructions causing the one or more processors to perform a method comprising: determining, by the electronic health record system and using Bayesian inference, for a patient, a first differential diagnosis list comprising a first subset of diagnoses selected from a plurality of potential diagnoses; determining, by the electronic health record system, for the patient, a first observation list comprising a first subset of observations selected from a plurality of potential observations, the first subset of observations selected based on having a largest utility for disambiguation of the first differential diagnosis list; dynamically adapting the anticipatory user interface to operate in a first mode, based on the first differential diagnosis list and the first observation list, wherein the operating of the anticipatory user interface in the first mode comprises: displaying, in the anticipatory user interface, the first differential diagnosis list to a physician; displaying, in the anticipatory user interface, for each diagnosis of the first subset of diagnoses, a current probability that the diagnosis accounts for previously made observationsupon the patient; displaying, in the anticipatory user interface, the first observation list to the physician; displaying, in the anticipatory user interface, for each observation on the first observation list, a current utility of the observation to distinguish among the first subset of diagnoses in the first differential diagnosis list; after displaying the first observation list to the physician, receiving, by the anticipatory user interface, a first ambiguous input from the physician; resolving the first ambiguous input by probabilistically determining an association of the first ambiguous input with at least one particular observation of the first observation list; and obtaining at least one value for the at least one particular observation and enter the at least one value into an electronic health record of the patient.

[0004] Other aspects of the invention will be apparent from the following description and the appended claims.BRIEF DESCRIPTION OF DRAWINGS

[0005] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures may be represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

[0006] FIG. 1 provides a block diagram of the components of an embodiment of the system.

[0007] FIG. 2A provides an example of the structure of a database derived from the objective electronic health records of observations in patients for whom diagnoses have been determined previously, in accordance with some embodiments.

[0008] FIG. 2B provides a flowchart describing an example method for generating and updating a diagnoses statistics database, in accordance with some embodiments.

[0009] FIG. 2C provides a flowchart describing an example method for Bayesian inference engine, in accordance with some embodiments.

[0010] FIG. 3 A provides an example of the structure of a set of matrices that holds information about the utility of various observations for discriminating among various diagnoses, in accordance with some embodiments.

[0011] FIG. 3B provides a flowchart describing an example method for computing observation utility matrices, in accordance with some embodiments.

[0012] FIG. 3C provides a flowchart describing an example method for Bayesian exploration to determine an observation list, in accordance with some embodiments.

[0013] FIG. 4A provides an example of the information displayed to the physician during use of the system, in accordance with some embodiments.

[0014] FIG. 4B provides a flowchart describing an example method for dynamically adapting an anticipatory user interface of an electronic health record system, in accordance with some embodiments.

[0015] FIG. 5 provides a flowchart describing an example method for reminding and / or empowering the physician to select and make observations that are relevant to a selected diagnosis, in accordance with some embodiments.

[0016] FIG. 6 provides a flowchart describing an example method for discounting observations related to previous or concurrent diagnoses to determine the true probability of other diagnoses, in accordance with some embodiments.

[0017] FIG. 7 provides the structure of a narrative note generated by the narrative note generator.

[0018] FIG. 8 provides an example of a computing system in accordance with some embodiments.DETAILED DESCRIPTION

[0019] Specific embodiments of the disclosure will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.

[0020] In the following detailed description of embodiments of the disclosure, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0021] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements, and is not to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before”, “after”, “single”, and other such terminology. Rather, the use of ordinal numbers is to distinguish between the elements. By way of an example, a firstelement is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

[0022] The present disclosure relates to a data-entry system that anticipates the observational actions of the observer so that the results of the observation may be entered automatically, efficiently and accurately into an objective record. A large database of such objective records representing many different causes of malfunction and their associated observations is analyzed statistically according to Bayesian Inference, which determines the probability of each diagnosis given the value of various observations, and Bayesian Exploration, which mimics the thought processes of an experienced observer in deciding which observations are most useful to make next, herein described as the utility of each observation. The actions of the observer may then be identified and recorded in the context of a short list of most useful and likely observations. Bayesian Inference and Bayesian Exploration are able to operate on a database of virtually unlimited size, comprising experience with a much larger number of records of malfunctions, their causes and the observations in those records, greatly exceeding the life experience of any individual person.

[0023] In one embodiment of the disclosure for medical use, the observer is a physician, the object is a patient, the diagnosis is a cause of a complaint or symptoms experienced by the patient, and the observations are derived from procedures such as a medical history, a physical examination, and various laboratory tests and specialty consultations. In one embodiment, the actions of the physician are signaled to the data-entry system by speech recognition of words spoken by the clinician and / or patient. In the case of self-diagnosis, a single person may act as both the patient and the physician. In some fields of healthcare, the role of the physician described herein may be performed by a dentist, physical therapist, chiropractor, or other caregiver. Embodiments as disclosed may be applied similarly to any complex object, however, such as an engineered machine, in which case the observer may be a service technician, the object may be a machine, the diagnosis may be a cause of a present or incipient malfunction of the machine, and the observations may be derived from operational or test data. In any of these embodiments, the utility of an observation may be weighted by the cost of the observation, as described in the prior art US Patents #9,690,906, #10,340,040 and #10,559,377, which are incorporated herein in their entirety by reference. In any of these embodiments, the objective records containing observations and diagnoses from many similar objects are collected automatically into the large database of electronic records that may be mined statistically to compute the utilityvalues that the data-entry system uses to create the short list of most useful and likely observations. Advantageously, the range of observations that a physician might make and enter into the electronic record is thereby extended from those with which they have personal experience to all those that have ever proven to be useful in the large database of records from many different physicians and patients. Embodiments of the disclosure take advantage of the likelihood that the probability of an experienced physician selecting an observation to make is related to the current utility of that observation for distinguishing among possible diagnoses. This has similarities to the autofill function used for typed entries into forms and messages but provides much more appropriate options because the Bayesian Exploration engine that selects them is constantly recomputing their utility based on the totality of the information already entered into the patient’s electronic health record (EHR).

[0024] Specific embodiments of the disclosure will now be described in detail with reference to the accompanying figures. In the following detailed description of embodiments of the disclosure, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well- known features have not been described in detail to avoid unnecessarily complicating the description.

[0025] In the following description of FIGs. 1-8, any component described with regard to a figure, in various embodiments of the disclosure, may be equivalent to one or more like-named components described with regard to any other figure. For brevity, descriptions of these components will not be repeated with regard to each figure. Thus, each and every embodiment of the components of each figure is incorporated by reference and assumed to be optionally present within every other figure having one or more like-named components. Additionally, in accordance with various embodiments of the disclosure, any description of the components of a figure is to be interpreted as an optional embodiment, which may be implemented in addition to, in conjunction with, or in place of the embodiments described with regard to a corresponding like-named component in any other figure.

[0026] FIG. 1 shows an overview of the various processes whereby a patient (102) with an undiagnosed condition is examined, diagnosed and documented by a physician (110) in EHR (112) using a system for identifying and entering observations that may be useful for the diagnosis of medical conditions, hereinafter referred to as system (100), in accordancewith one or more embodiments of the disclosure. Processes shown as circles are described in detail in subsequent figures. The observations may be recorded as either Boolean values (1 or 0, corresponding to presence or absence of a feature such as male gender or itching sensations) or continuous values (integer or rational numbers such as age, pain on a scale of 1-10, or concentration of glucose in the blood). While both types of observations may be used by the processes described herein, they may need different handling as described in the detailed descriptions below.

[0027] Turning to FIG. 1, system (100) may initially receive and collect basic patient information such as demographics, chief complaint, vital signs, current medications, etc. through intake process (120), which enters the collected information into the patient’s EHR (112). One EHR (112) may exist for each patient, each complaint of a patient to be diagnosed, each encounter with a patient or each diagnosis given to a patient. Formal coding systems may be relied upon to give each potential diagnosis and / or observation in an EHR (112) a unique and machine-searchable identifier so that individual EHRs may be combined into a large database that can be processed by computerized software algorithms. Intake process (120) may include the patient responding to a questionnaire, a paramedical technician performing tests, a clerk entering data from an external source, and / or a physician asking standard questions, in accordance with common clinical practice.

[0028] Subsequently, in accordance with one or more embodiments of the disclosure, system (100) uses Bayesian Inference engine (140) to operate on the information currently in the patient’s EHR (112) and information in a diagnoses statistics database (130) compiled from the collected EHRs of many patients with many diagnoses as described below. The diagnoses statistics database (130), in accordance with an embodiment of the disclosure, thus provides sufficient depth and breadth of information to capture what actually has occurred with large numbers of patients in medical practice. Various filters, cleaners, translators, synonyms, statistical analyses and other computerized methods for individual and / or institutional records may be used in order to create, maintain and / or access the diagnoses statistics database (130). Bayesian Inference engine (140) computes a differential diagnosis list (142) consisting of the current probabilities of various diagnoses as a cause of the patient’s condition, as described in more detail below. The system then uses a Bayesian Exploration engine (150) to determine the utility of possible next observations, based on statistical data obtained from the diagnoses statistics database (130) and the differential diagnosis list (142) produced by the Bayesian Inference engine. An extensive description ofthe steps performed by the Bayesian Exploration engine to determine a utility weighted list of possible observations, hereinafter observation list (152), is subsequently provided with reference to FIGs. 3A-B. Observations identified as useful by the system may be any type of clinical action, including questions asked in a medical history, physical examination, laboratory tests, imaging, specialty consults, and therapeutic trials. It may be helpful to organize the observations on the observation list into sublists according to type of clinical action. Observations may be composite, consisting of multiple data values for related items, which may be Boolean (yes / no), continuous variables, and / or discrete options. Advantageously both the most probable items on the differential diagnosis list (142) and the most useful items on the observation list (152) as well as the current contents of EHR (112) may be provided by display device (160) to physician (110) to alert them to possibilities that they might have overlooked. Display device (160) may be a video monitor for text and / or graphics or a loudspeaker or headphone driven by a speech synthesizer, or any combination of devices that can convey information from system (100) to physician (110) and / or patient (102).

[0029] Still referring to FIG. 1, the actions of physician (110) are recorded by input device (165), which may include one or more of a microphone, video camera, keyboard or physical pointing device such as a computer mouse or touchscreen. In one embodiment, input device (165) includes a microphone and natural language processor for identifying spoken words. The actions of the physician are interpreted by observation identifier (170), which advantageously considers the probability that the physician has performed one of the observations in observation list (152) or has accepted a diagnosis (172) from differential diagnosis list (142). Based on that interpretation, the system then may enter data from the observation into the appropriate part of the EHR (112) or generate a request (175) for the observational data to be returned (176) from an external source, perhaps with some delay (denoted by dashed line in FIG. 1). Display device (160) and input device (165) may be used to create a dialogue between system (100) and physician (110) and / or patient (102), such as to propose and seek confirmation of the desired next action. For example, if the physician says “Let’s get a picture of that” and the system has identified that an observation that has high utility in the observation list requires a CT scan, then the system could use speech synthesis to advise the physician that “I will order a CT scan” and then await a confirmatory “Yes” or a countermand such as “No, get an MRI scan”.

[0030] If and when physician (110) accepts a diagnosis (172), then system (100) may use additional processes identified at the bottom of FIG. 1 to address additional requirements described below in connection with FIGs. 5-7. One of these processes may be phenotype completion tool (180), which provides opportunities for physician (110) to make additional observations that system (100) has identified as indicative of the accepted diagnosis. Another process may be multidiagnosis engine (190), whereby an accepted diagnosis is removed from consideration and the probabilities of other, concurrent diagnoses are recomputed. Yet another process may be narrative note generator (197), by which information contained in EHR (112) is reconfigured into a report suitable for transmission to another physician.

[0031] The structured nature of EHRs (112) described herein is designed to facilitate the operation of the various engines described in this specification, but it may not be deemed suitable for communicating with other clinicians. Clinicians traditionally generate a narrative note summarizing their clinical encounters, observations, diagnoses, treatments and outcomes. This narrative note may be generated by many means, including hand-writing into a template form, keyboarding free text into a computer, dictation and manual transcription, or dictation and natural language processing by a computer. Narrative notes generated after clinical encounters have been found to be prone to errors of memory or omission, particularly for observations that may be inconsistent with the physician’s diagnosis and, therefore, important for identifying erroneous diagnoses. Narrative note generator (197) may be used to convert and summarize structured data in the patient’s EHR (112) into the form of a textbased narrative note that may be added to the patient’s EHR to facilitate communication with other clinicians, as described in FIG. 7. This may be accomplished by natural language generation, a form of artificial intelligence based on deep-learning neural networks.

[0032] FIG. 2A shows one useful structure for diagnoses statistics database (130) according to some embodiments. Each diagnosis Dnoccupies one row and each observation Oi occupies one column. Each cell at the intersection of a row and column holds one statistical summary of all values of that observation recorded with that diagnosis. Each statistical summary consists of three numbers. If the observation is a Boolean type as illustrated for Oi, the first number is a zero indicating Boolean, the second number is the number of zero values NOoi.Dn and the third number is the number of 1 values Nloi.Dn. If the observation is a continuous type as illustrated for O2, the first number is the number of values obtained N (always nonzero), the second number is the mean poi,Dn and the third is thestandard deviation ooi,Dn. Many other structures for such a database will be apparent to those familiar with informatics and are within the scope of the disclosure.

[0033] Diagnoses statistics database (130) may be implemented using any format suitable for the storage of such data, such as any type of hierarchical, relational and / or object oriented collection of data. The diagnoses statistics database (130) may be hosted in nonvolatile memory, e.g., on a hard disk drive, a redundant array of independent disks (RAID), network attached storage (NAS), cloud storage, etc. Further, at least some of the content of the diagnoses statistics database (130) may alternatively or in addition be stored in volatile memory, e.g., Dynamic Random-Access Memory (DRAM), Synchronous DRAM, SDR SDRAM, and DDR SDRAM. Those skilled in the art will recognize that the diagnoses statistics database (130) may be under the administration of various legal entities such as healthcare providers, health insurance providers, government agencies, etc. Advantageously, as described in FIG. 2A, diagnoses statistics database (130) consists of deidentified data reflecting only the cumulative probabilities of obtaining particular results from various observations made in patients with various diagnoses, thus avoiding many regulatory obstacles and privacy considerations associated with health data. The structure of diagnoses statistics database (130) described herein includes data that are sufficient to perform the various computational processes of system (100). Various transformations of these data may achieve the same functionality and are included in the scope of the disclosure. Additional information may be stored in system (100) such as intermediate or derivative computations that facilitate or accelerate real-time computations during a clinical encounter, or more detailed information from EHRs (112) such as may be useful for correcting erroneous information that has been incorporated into diagnoses statistics database (130). Some of the information stored in system (100) and / or the calculations performed by processes in system (100) may be in physically different locations, such as a cloud network or a central server or local computers at the point of care. Such variants of system architecture are well-known to information technologists and are within the scope of the disclosure. Operations used to establish and / or update the diagnoses statistics database (130) are described below in reference to FIG. 2B.

[0034] FIG. 2B provides a computational process for generating and updating diagnoses statistics database (130) with data from an EHR (112). The subsequently described operations may be performed by the process (195) for adding records to the diagnoses statistics database. These operations may be performed (a) initially to establish the diagnosesstatistics database (130) or (b) whenever an update of the diagnoses statistics database (130) is necessary or desirable. The newly entered data may thus become applicable when electronic health records of future patients are processed in accordance with embodiments of the disclosure. When initially performed, the method of FIG. 2B may be executed repeatedly for multiple or many EHRs (112). Later executions may be performed based on a set schedule to ensure that the diagnoses statistics database (130) remains accurate in view of changes to the EHRs (120). Later executions may also be triggered by certain events, for example, a change in an EHR that would affect the content of the diagnoses statistics database (130). Such changes may be a result of input provided by the physician (110) or external sources.

[0035] In Step 202, the EHR to be processed is selected.

[0036] In Step 204, a diagnosis is obtained Dnfrom the selected EHR.

[0037] In Step 206, an observation value Oi from the selected EHR. I

[0038] In Step 210, it is determined whether Oi is Boolean or continuous data type. If continuous, the method may proceed with the execution of Step 212. If continuous, the method may proceed with the execution of Step 216.

[0039] In Step 212, the count N, mean poi.Dn, and standard deviation ooi,Dn are updated based on the newly obtained Oi.in the diagnoses statistics database (130)

[0040] In step 214 either NOoi.Dn or Nloi,Dn are incremented in the diagnoses statistics database (130), depending on the observation value.

[0041] In Step 220, if one or more observation values to be processed are remaining in the EHR, the execution of the method proceeds with Step 206 to process the remaining observation value(s). If no observation values are remaining, the execution of the method proceeds with Step 222.

[0042] In Step 222, if one or more diagnoses to be processed are remaining in the EHR, the execution of the method proceeds with Step 204 to process the remaining diagnosis(-es).

[0043] While the method of FIG. 2B describes an updating of the diagnosis statistics database, the updating may include an addition of new elements, for example, if no statistics for a diagnosis or an observation found in the EHR currently exist in the diagnosis statistics database. In this case, the diagnosis statistics database may be updated to include statistics for the diagnosis / observation.

[0044] FIG. 2C provides a computational process for Bayesian Inference engine (140), which operates upon diagnoses statistics database (130). Bayesian Inference is a well- known, incremental, statistical process of refining estimates of the probabilities of all possible causes of observational data as each new datum is provided. The prior probability of each cause before the datum is provided is converted into a posterior probability based on the new datum’s value and the probabilities of that value having been observed in a large set of experiences of various causes. That posterior probability then becomes the prior probability for the recomputation of probabilities after a subsequent datum is provided. For the medical embodiment discussed here, the data are clinical observations, and the causes are medical disorders identified as diagnoses. Before any data have been provided, the initial prior probability of a patient having a given diagnosis P(Dn) is simply the number of occurrences of that diagnosis in diagnoses statistics database (130) divided by the total number of occurrences of all diagnoses in diagnoses statistics database (130). The sum of all probabilities P(Dn) must always equal one. As each new observation Oi is obtained, the probability of each diagnosis Dnmay be updated according to Bayes’ Theorem:

[0045] The denominator of Bayes theorem P(Oi) normalizes the weighted probability in the numerator by the current probability of encountering data value Oi, which is the sum of all weighted probabilities in the numerator for all diagnoses:

[0046] When Bayes’ Theorem is applied for the next available data item, both its numerator and the sum in the denominator may be recomputed.

[0047] Note that which observations are available during a given work-up and in what order is completely arbitrary. In fact, it is an objective of Bayesian Exploration (see below) to dynamically reorder the collection of data items so that the most useful observations for establishing the diagnosis are generally obtained next during the work-up and items that are irrelevant or superfluous may be skipped entirely. The probability of observation Oi in a patient with diagnosis Dnis P(Oi|Dn), which does not change during a given work-up. These probabilities may be computed off-line and stored for rapid access during clinical encounters. P(Oi|Dn) is computed only for those EHR records for which a data value is available, so blankvalues in the database are not a problem. Data items that do not occur or are skipped during a given work-up do not trigger a Bayesian inference calculation, and therefore do not cause a problem.

[0048] The mathematical implementation of Bayes Theorem depends on the type of observation as defined above. It is relatively straightforward for Boolean type observation but depends on probability density functions rather than probability of obtaining exact values for continuous data.

[0049] For Boolean data type, the conditional probability in the numerator P(Oi|Dn) must be computed from all records with diagnosis Dn. The normalizing value P(Oi) in the denominator is the sum of the conditional probabilities for all diagnoses P(Oi|Dn), each weighted by its current (Bayesian prior) probability P(Dn). For examples, suppose the question of patient gender is the first data item encountered (Oi) and it assigns value 0 for male and 1 for female. Suppose there are equal numbers of males and females in the database. Suppose there are only two diagnoses in the database: half the patients have Di and half have D2 but 80% of the patients with diagnosis Di are female and only 20% with diagnosis D2 are female. Before any data are available, the prior probability of a new patient having either diagnosis is 0.5. If the new patient is female (Oi=l), the prior 0.5 probability of diagnosis Di is multiplied by P(Oi|Dn) = 0.8 and divided by the weighted sum of 0.8 x 0.5 + 0.2 x 0.5 = 0.5 so its posterior probability P(Oi|Dn) is now 0.8. The posterior for diagnosis D2 is 0.5 x 0.2 / 0.5 = 0.2. Note that the probabilities of all updated diagnoses still add to 1. Note that if the observation Oi has value 0 (i.e., patient is male in this example), then the probability must be computed for the “not true” condition “'P(OilDn) = 1- P(Oi|Dn).

[0050] For continuous and presumably normally distributed data, the equation below has been adapted from Fishel, J. A., & Loeb, G. E. (2012). Bayesian exploration for intelligent identification of textures. Frontiers in Neurorobotics, 6(4). doi: 10.3389 / fnbot.2012.00004, which describes both Bayesian inference and Bayesian exploration as applied to the related problem of efficiently identifying the most likely identity of an object from incremental acquisition of data via sequential exploratory movements performed upon that object. For continuous data, the probability of obtaining a specific value is essentially zero if the scale has infinite resolution. The updating of each prior probability P(Dn) is accomplished by the ratio of two probability density functions, as described in eq. 1 of that paper. The probability P(Dn) cannot be computed directly but probability densityfunctions p that are proportional to P may be computed from the means pi and standard deviations G, of the distributions according to the following equation:

[0051] Note that PDF is a density with a value that is l / (unit of measure) for the data Oi but these unknown scaling terms cancel out because they are in both the numerator and the denominator of Bayes’ Theorem.

[0052] Now referring to the flowchart of FIG. 2C, steps that perform the described Bayesian inference are shown.

[0053] In Step 252, an observation Oi is obtained. The observation Oi is a previously made observation and may be obtained from the patient’s EHR.

[0054] In Step 254 a diagnosis Dnis selected from the diagnoses statistics database (130).

[0055] In Step 256 a prior probability of that diagnosis P(Dn) is obtained.

[0056] In Step 258 the posterior probability P(Dn| Oi) is computed according toBayes Theorem as described above.

[0057] In Step 260 if one or more diagnoses to be processed according to Steps 256 and 258 are remaining, the method may proceed with Step 254 to select another diagnosis.

[0058] In Step 270, the differential diagnosis list (142) is selected for display according to their posterior probabilities P(Dn| Oi). A set number of diagnoses with a highest posterior probability may be included in the differential diagnosis list, or all diagnoses associated with posterior probabilities that exceed a set threshold may be included.

[0059] As described below, the physician may then choose another observation, and obtain a result (e.g., a value) of the observation. Subsequently, the method of FIG. 2C may be repeated, and a new differential diagnosis list and a new set of observations and costs may be obtained under consideration of the result.

[0060] FIG. 3A shows a structure for observation utility matrices (137), which are used by the computational process in Bayesian Exploration engine (150) illustrated in FIG. 3B. One observation utility matrix U(Oi) is computed for each observation Oi in diagnoses statistics database (130). This computation may be performed when the Bayesian Exploration engine (150) is activated or when the process to add record to database (195) is activated or at regular or occasional update intervals decided by system (100). The set of all currentobservation utility matrices (137) may be stored in diagnoses statistics database (130) or other non-transitory memory storage, thereby facilitating the computations of Bayesian Exploration engine (150) that are performed frequently during the clinical work-up of the patient. The elements of each cell Uoi,Dm,n in each observation utility matrix (137) have a value of one minus the degree of overlap DOom,n between the values obtained for that observation in patients with diagnosis Dmand patients with diagnosis Dn. For Boolean data, DOoi,Dm,n is simply the absolute difference between the probabilities P(Oi | Dm) and P(Oi | Dn), so

[0062] For continuous data, DOoi,Dm,n may be computed from the Bhattacharyya coefficient for overlap between two normal distributions:

[0063] Referring to FIG. 3B, utilities matrices engine (151) computes observation utility matrices (137) according to the following steps.

[0064] In Step 302, an observation Oi is selected. Oi may be selected from many observations, for example from all observations available across all EHRs that are available for processing, or a subset of these observations.

[0065] In Step 304, a pair of two different diagnoses, Dnand Dm, is selected. These diagnoses Dnand Dmmay, again, be selected from many diagnoses, for example from all diagnoses available across all EHRs that are available for processing, or a subset of these diagnoses.

[0066] In Step 310, the utility Uoi,Dm,n of observation Oi to distinguish between diagnoses Dmand Dn, which is one minus the degree of overlap DOoi,Dm,n as described above.

[0067] In Step 312, Uoi,Dm,n is entered into the corresponding cell of utility matrix Uoi,Dm,n, as illustrated in FIG. 3 A.

[0068] In Step 322, if one or more additional pairs of Dnand Dmare feasible, the execution of the method may proceed with Step 304 to process these additional pairs in Steps 310 and 312.

[0069] In Step 322, if one or more additional pairs of Dnand Dmare feasible, the execution of the method may proceed with Step 304 to process these additional pairs in Steps 310 and 312.

[0070] In Step 324, if one or more observations are remaining, the execution of the method may proceed with Step 303 to perform the operations of Steps 304-322 for these observations.

[0071] Referring to FIG. 3C, Bayesian Exploration engine (150) computes total probability-weighted utility TPUoi for each observation Oi. The result may be displayed in the form of an observation list, further discussed below.

[0072] In Step 352, an observation Oi is selected. Oi may be selected from many observations, for example from all observations that are statistically represented in the observation utility matrix.

[0073] In Step 354 a diagnosis Dnis selected. Dnmay be selected from many diagnoses, for example from all diagnoses that are statistically represented in the observation utility matrix.

[0074] In Step 356 another diagnosis Dm. that is different from Dnis selected. Dmmay be selected from many diagnoses, for example from all diagnoses that are statistically represented in the observation utility matrix.

[0075] In Step 360 the probability weighted utility PUoi,Dm,n is computed, e.g., by multiplying the utility value Uoi,Dm,n in each cell of observation utility matrix Uoi (137) by the current Bayesian probabilities of the two diagnoses that denote the row and column of the cell, P(Dm) and P(Dn).

[0076] In Step 362, all values in the matrix of probability weighted utility PUoi,Dm,n are summed to obtain the total probability-weighted utility TPUoi for each observation Oi. Note that Uoi is symmetrical, and the diagonal cells are necessarily zero, which permits a more efficient computation of the sum TPUoi than operating on the entire matrix U(Oi).

[0077] Steps 372, 374 and 376 complete loops through all diagnoses Dmand Dnand all observation Oi, respectively.

[0078] In Step 380, the items comprising observation list 152 are selected according to their total probability -weighted utility TPUoi. Additional details are provided in reference to FIG. 4A.

[0079] FIG. 4A shows a screen capture for an embodiment of the disclosure in which display device (160) is a computer monitor visible to physician (110). The current differential diagnosis list (142) with the probability of each as computed by Bayesian Inference engine (140) is displayed at the upper right, indicated by the dashed box. The current observation list(152) is displayed under it, ordered according to the current total probability -weighted utility of each observation TPUoi or its sum for a group of observations.

[0080] Observation list (152) may be advantageously ordered according to descending values of total probability-weighted utility TPUoi for individual observations Oi and only the higher valued items may be sent to display device (160). It may be advantageous for closely related observations, such as the location and appearance of a skin lesion, to be grouped and displayed together to the physician in an observation entry panel along with options to set observations not specifically addressed by the physician to default, negative or unobserved status. It may be useful to design the process of entering grouped observations in a manner that encourages physician (110) explicitly to assert negative Boolean observations by requesting data for various observations, particularly those that have consistently negative values in some but not all diagnoses in the current differential diagnosis list (142). In some cases, values for a group of observations may be obtained in a single observational procedure such as an imaging study, a complete blood count, or a consultant’s report. For such groups, observation list (152) may point to a group whose total probability -weighted utility may reflect the sum of the TPUoi values of all items in the group. System (100) may employ simple criteria to decide how many observations or groups of observations to include in the display of observation list (152), such as a fixed number of observations, all observations with a TPU value over some threshold, observations of a particular nature such as questions in a medical history, or other considerations or combinations of considerations that may change over the course of a clinical encounter.

[0081] In the example as shown, the physician has used input device (165) to navigate to a group of observations identified as the “General Location of Problem” in the center observation entry panel via observation identifier (170). Observation identifier (170) may use subsequent observations of the physician as conveyed via input device (165) to enter observational data into EHR (112). This particular group of observations is one of many groups to which the physician might navigate, as indicated in the manual navigation panel at the left, but advantageously, observation list (152) provides observation identifier (170) with the most likely choices that the physician will make, improving the speed and accuracy of the navigation and data entry steps enabled by observation identifier (170). These features and the underlying methods may be provided by the anticipatory user interface (155), as subsequently discussed in reference to FIG. 4B.

[0082] FIG. 4B shows an example method for dynamically adapting an anticipatory user interface (155) of an EHR system (100), in accordance with some embodiments. Using the method, the actions of the physician may be interpreted by the system so that navigation and data entry may be performed efficiently and accurately. In one embodiment, physician (110) collects observational data from patient (102) while input device (165) monitors the interactions via a microphone. Observation identifier (170) may use natural language processing software that identifies words and phrases and that computes the confidences that they signify observational data intended for various destinations in the patient’s EHR (112) or represent requests for observational data that must be obtained from external sources such as laboratory testing, radiographic imaging, surgical biopsy or specialty consultation. The destinations or requests and their probabilities may be computed by a neural network trained upon sets of data from many clinical interactions. It is likely that many destinations or external sources might have similarly high probabilities, particularly if the identified words and phrases are fairly general and conversational, as would occur naturally during a clinical encounter. For example, a physician might ask “Is this painful?” while manipulating some body part on the patient, but there would be many places in the EHR where pain or tenderness of different body parts subjected to different manipulations could be entered as observations. Observation list (152) advantageously includes the utility of each of those observations, which may be related to the probability that the physician is making each of those observations. Observation identifier (170) may multiply the confidence of the destinations identified by the natural language processing software by the utilities of the items on observation list (152) to identify the most likely destination in the patient’s EHR (112). Display device (160) may be used to show the physician the most likely destination, which the physician may accept by continuing to make the observation and enter the data via input device (165) or may be countermanded by a different signal to the input device. For example, the physician might say “3+ tenderness” and the observation identifier would compute from the contents of observation list (152) and their utilities that this most likely refers to the patient’s left temporomandibular joint. Alternatively, the physician might note from display device (160) that the system is expecting a value for joint pain, but the physician instead wishes to enter a value for masseter muscle tenderness, another highly useful observation, in which case the physician might say “3+ muscle tenderness” to avoid the ambiguity. Similarly, a physician might initiate a request for external data by stating “Let’s get a picture of that,” which might signify a need for a photographic image, a radiographicimage such as an x-ray, CAT scan or MRI, or a histological analysis of a tissue sample. Again, the ambiguity may be resolved by multiplying the probability of each potential request by the sum of the total probability-weighted utilities TPU(Oi) of the observations that might be thereby obtained.

[0083] Consider the following specific example: Based on what is shown in the Observation Entry Panel of FIG. 4A, the physician has identified that the problem is in the lower jaw. If the physician says “Let’s get a picture of that,” the ambiguity regarding “that” may be resolved by the observation that the patient’s problem relates to the lower jaw and not other body parts. If the physician has entered another observation that there is a palpable hard tumor in the lower jaw, the ambiguity regarding “picture” may be resolved by the utility of a CT scan for diagnosing various boney lesions. System (100) may then use display device (160) to convey to the physician its intention to order a CT scan of the lower jaw, which the physician can simply accept or countermand.

[0084] Advantageously, request generator (175) may include in the request a list of the observations from observation list (152) that have the highest total probability -weighted utility, thereby providing a template for returned data (176) that is already coded for entry into the patient’s EHR (112). For example, if a high utility observation is whether or not the patient has a pleural effusion and that can be observed by requesting a chest X-ray, then request generator (175) may include in the request that specific observation and the radiologist may then return data (176) that provides a value for that specific observation, which value may then be entered automatically into the patient’s EHR (112).

[0085] Turning to FIG. 4B, in Step 400, a differential diagnosis list is determined for the patient. The differential diagnosis list may be determined using Bayesian inference based on operations as previously described, e.g., in reference to FIG. 2C. The differential diagnosis list includes a subset of diagnoses selected from potential diagnoses. The potential diagnoses may be all diagnoses (or at least some of all diagnoses) for which Bayesian exploration, e.g., as described in reference to FIG. 3C has been performed.

[0086] In Step 402, an observation list is determined for the patient. The observation list includes a subset of observations selected from potential observations. The potential observations may be all observations (or at least some of all observations) for which Bayesian exploration, e.g., as described in reference to FIG. 3C has been performed. The subset of observations may be selected based on having a largest utility for disambiguation of the first differential diagnosis list, e.g., as described in reference to FIG. 2C.

[0087] In Step 410, the anticipatory user interface is dynamically adapted to operate in a particular mode. The mode is based on the differential diagnosis list and the observation list. The operation of the anticipatory user interface in the mode has implications on the results produced by the subsequently described Steps 412-424. Specifically, depending on the operating mode of the user interface (as determined based on the differential diagnosis list and the observation list), display output provided to the physician may be completely different, and / or input received from the user may be interpreted completely differently. Importantly, by dynamically adapting the anticipatory user interface in this manner, the physician receives more relevant information, and / or input provided by the physician is interpreted based on the current context, thereby improving the accuracy of such interpretations and / or facilitating the physician’s task. Additional details are subsequently provided.

[0088] In Step 412, the differential diagnosis list as determined in Step 400 is displayed to the physician. An example is provided in FIG. 4A.

[0089] In Step 414, for each diagnosis of the subset of diagnoses on the differential diagnosis list, a current probability is displayed. An example is provided in FIG. 4A. The probability is a probability that the diagnosis accounts for previously made observations upon the patient and may be obtained as described in reference to FIG. 2C.

[0090] In Step 416, the observation list as determined in Step 402 is displayed to the physician. An example is provided in FIG. 4A. As previously noted, the observation list that is displayed may include one or more sublists in which the observations in each sublist may be obtained by one type of interaction with the patient.

[0091] In Step 418, for each observation on the observation list, a current utility of the observation to distinguish among the subset of diagnoses in the differential diagnosis list is displayed. An example is provided in FIG. 4A. The current utility may be obtained as described in reference to FIG. 3C. Utility values as displayed for observations may consider expected costs of the observations.

[0092] In Step 420, after displaying the observation list to the physician, an ambiguous input is received from the physician. The input may be obtained via input device 165. In one example, the input is speech recorded by a microphone and processed by a natural language processor The ambiguity may be a result of an input that, analyzed in isolation, cannot be fully resolved. Examples are provided in reference to FIG 4A. Consider the previously introduced request “let’s get a picture of that”. The request is ambiguous inthat it does not specify (a) the type of picture to be taken, and (b) the target of the picture to be taken.

[0093] In Step 422, the ambiguous input is resolved by probabilistically determining an association of the ambiguous input with at least one particular observation of the observation list, as previously described.

[0094] In the example of the ambiguous input including speech processed by a natural language processor, the probabilistic processing may involve multiplying a confidence of an output of the natural language processor (based on the processing of the speech) times the utilities of the observations on the first observation list to obtain one product for each observation on the observation list, thereby considering not only the uncertainty associated with the observation list but also the uncertainty associated with the processing of the speech.

[0095] After the resolution of the ambiguous input, the at least one particular observation may be performed, either by the physician or by a third party.

[0096] In Step 424, at least one value for the at least one particular observation is received (e.g., from the physician or from elsewhere), and the anticipatory user interface may then automatically enter the at least one value into the EHR of the patient. When the source of the value for an observation is a source external to the electronic heath record system (e.g., another clinic), the anticipatory user interface is configured to send a request to the external source for the observation to be performed, and to receive the value from the external source. The anticipatory user interface may then automatically enter the value into the electronic health record of the patient.

[0097] As indicated by the return arrow from Step 410 to Step 400, the operations shown in FIG. 4B may be performed in a loop. With each iteration, the dynamic adaptation of the anticipatory user interface may result in different outcomes. Specifically, for example, the differential diagnosis list and associated probabilities displayed to the user may be different (Steps 412, 414), the observation list and associated utilities maybe different (Steps 416, 418), and the resolution of the ambiguous input may result in a different outcome (Steps 420-424).

[0098] As the operations are repeated, e.g., in a first repetition or a second repetition of the loop, the physician may feel increasingly confident about a diagnosis on the current differential diagnosis list (or another diagnosis). Accordingly, the physician may choose to provide their choice of a diagnosis as an input. Such an input, analogous to the input associated with the selection of an observation, may be ambiguous. In this case, theresolution of the ambiguous input (Step 422) may involve identifying a diagnosis on the differential diagnosis list rather than an observation on the observation list. Once the diagnosis is identified after addressing the ambiguity, a list of treatment options for the diagnosis may be displayed to the physician, and an input from the physician may indicate the selection of one or more of the treatment options.OPTIONAL PROCESSES FOR ELECTRONIC HEALTH RECORDS

[0099] FIG. 5 shows a computational process for phenotype completion tool (180). As physician (110) enters data from the highest utility observations in observation list (152), the probabilities of diagnoses in differential diagnosis list (142) may approach the maximal value of 1 for the one diagnosis that most accurately reflects the patient’s condition while the probabilities of all other diagnoses drop toward zero. At that point, all observations not yet performed and entered have minimal total probability -weighted utility TPUoi because no uncertainty of diagnosis remains to be resolved. As described in the text referring to FIG. 3C, each cell of each observation utility matrix (137) is multiplied by the current probabilities of the two diagnoses representing the row and column address of the cell, resulting in very small values. For a typical presentation of the high probability diagnosis, this may occur before the physician has entered at least some of the observations that are normally associated with that diagnosis. While the unmade observations may no longer be necessary, it is desirable that the patient’s EHR reflect accurately the complete set of observations typically associated with the final, selected diagnosis, which set is known as a phenotype. The observations that constitute the phenotype for a given diagnosis may be determined from textbooks, journal articles, expert consultants or extracted from the diagnoses statistics database (130) by identifying those observations that have the highest probability of having a particular value in those patients who have the selected diagnosis. These observations may be stored in a phenotype list (192). Separate phenotype lists may exist for different diagnoses. Phenotype completion tool (180) prompts the physician via display device (160) to consider making those observations that have not yet been performed from the phenotype for the selected diagnosis. Physician (110) may navigate immediately to the relevant portion of EHR (112) by selecting an observation from the list of observations that have not yet been performed. Observation identifier (170) may then assume that the subsequent physician’s actions conveyed to it from input device (165) constitute the value of the selected observation to be entered into EHR (H2).

[0100] Referring to FIG. 5, in Step 502, an accepted diagnosis Dnis obtained. The accepted diagnosis may be obtained based on confirmation of a diagnosis by the physician (110), e.g., by the physician selecting one of the diagnoses on the differential diagnosis list (142) or by entering or selecting a different diagnosis.

[0101] In Step 504, the phenotype list is obtained for Dn. The phenotype list may have been populated with the above-described phenotype information as discussed.

[0102] In Step 510, the phenotype list that identifies the observations that have not been entered is displayed to the physician (110) via display device (160). Based on the list displayed to the physician, the physician may then choose observations for which they intend to enter values or to exit phenotype completion tool (180).

[0103] In Step 512, the process awaits an input, the input identifying an observation from the list displayed in Step 510 and providing a value for the observation.

[0104] In Step 520, if no such input is received, the execution of the method terminates. If such an input is received, the execution of the method proceeds to Step 522

[0105] In Step 522, the observation value is entered into EHR (112), and the process loops back to step 510 to display an updated phenotype list.

[0106] FIG. 6 shows the computational process for multidiagnosis engine (190), which may address two problems related to patients that have more than one active diagnosis. First, many of the diagnoses that can be found in standard, coded lists, such as available from the International Classifications of Diseases (ICD) from the World Health Organization, are not mutually exclusive. For example, there may be a diagnosis that accounts for general inflammation of a body part and another diagnosis that accounts for a specific cause of the inflammation, both of which represent valid diagnoses for a patient. If the observations entered into the EHR are consistent with two or more diagnoses, none of them will be able to approach a conclusive probability of 1 because the total probabilities of all diagnoses must add to 1. Second, if a patient has two or more independent conditions, then observations related to the diagnosis of one condition may interfere with the identification of the diagnosis of the other condition(s). Two or more diagnoses that are active in a single patient may interact in ways that cause the presentation of each to appear somewhat atypical. Because the interactions between observations are likely to be nonlinear, there is no simple way to separate these effects. The result is that no single diagnosis will achieve a high probability. Both of these problems may be mitigated by running a simulated diagnostic work-up of the patient after removing a selected diagnosis from differential diagnosis list (142). Theprinciple behind this is that only those observations will be considered that differentiate among the remaining diagnoses in differential diagnosis list (142) rather than considering those observations that had high utility for the selected diagnosis that is being discounted.

[0107] Referring to FIG. 6, in Step 602, a diagnosis to discount is received. The diagnosis to discount may be selected by the physician, either from previously accepted diagnoses in the patient’s EHR (112) or the newly accepted diagnosis (172) from the differential diagnosis list (142).

[0108] In Step 604, a copy of the patient’s EHR (112) is made and saved as reference health record (114).

[0109] In Step 606, all observations except those entered during intake process (120) are removed from EHR (112).

[0110] In Step 608, the probability of the diagnosis to discount is set to zero. To indicate the discounting, the discounted diagnosis may be displayed at the top of the list with a probability of zero.

[0111] In Step 610, the probabilities associated with the remaining diagnoses (the diagnoses other than the diagnosis to discounted) are recomputed. The recomputing may be performed using Bayesian inference as previously described. The updated differential diagnosis list (142) may be displayed to the physician.

[0112] In Step 612, the observation list is recomputed. The recomputing may be performed using Bayesian exploration as previously described. The updated observation list (152) may be displayed to the physician.

[0113] In Step 614, multidiagnosis engine (190) determines if the observation with the highest total probability weighted utility TPUoi in observation list (152) has a corresponding observation value in reference health record (114). If yes, then step 616 transfers that observation value into the patient’s EHR (112) and loops back to step 610. If no, then exit the multidiagnosis engine.

[0114] After exiting the multidiagnosis engine, the physician may continue with the work-up process using any of the various procedures described above, conveying their intentions via input device (165) and observation identifier (170). These procedures may include selecting, performing and entering a value for one of the observations on observation list (152), generating a request (175) for an observation to be performed externally, or accepting a diagnosis from differential diagnosis list (142). It is anticipated that the physician will use clinical judgment to avoid selecting observations or diagnoses on those lists that thephysician knows to be misleading as a consequence of interactions with the diagnosis to discount. When the clinician exits the work-up process, system (100) may employ process (195), described below and in FIG. 7, to add to diagnoses statistics database (130) data from the current EHRs (112 and 114).

[0115] FIG. 7 shows an outline of the output of narrative note generator (197). A narrative note, generated by the narrative note generator (197) may essentially be a human- readable version of the EHR. The double-underlined topics may be compiled directly from the coded observations that system (100) has entered into EHR (112). Advantageously, individual observations within each topic may be ordered according to the order in which they were entered by physician (110), which may reflect their total probability -weighted utility at the time when they were entered, hence their relevance to the diagnostic process. Treatment Prescribed may consist of items selected by physician (110) from a list of treatment options for the patient’s diagnosis, which list may be generated by clinical experts and / or statistical analysis of responses to treatment in other patients with the same diagnosis, as described above. The remaining items (Current Problem, Management Notes) provide an opportunity for the physician to insert free text, such as by keyboarding or dictation with human or machine transcription, that reflects clinical impressions and plans for follow-up that may not be represented in the coded observations available in the system. Advantageously, the output of narrative note generator (197) may be saved in the patient’s electronic health record (112) or other digital storage location.OPTIONAL REFINEMENTS TO DIAGNOSES STATISTICS DATABASE (130)

[0116] With reference to FIG. 2B, the computational processes to add information (195) from the patient’s EHR (112) to diagnoses statistics database (130) and to use it for Bayesian Inference may benefit from some refinements. As described above and in FIG. 2A, diagnoses statistics database (130) is organized such that, for each diagnosis ever entered, there is information regarding the probabilities of all patients with that diagnosis having a particular value of each observation. Advantageously, each EHR (112) and each reference health record (114) may contain one and only one accepted diagnosis and the values of all observations associated with that diagnosis.

[0117] For both Boolean and continuous data types, it is desirable to be able to update the cumulative values from many patients recorded in diagnoses statistics database (130) without maintaining a record of the individual values that contributed to the cumulativevalues. For Boolean data, probability P is simply the sum of the individual values divided by the number of values in the sum N. In the embodiment described in FIG. 2A, Boolean observations associated with a given diagnosis are recorded in diagnoses statistics database (130) as the number of zero values NOoi.Dn and the number of one values Nloi,Dn. Probability P for that Boolean observation is N1 / (N1+NO). A new Boolean data value (0 or 1) may simply be added to the appropriate count NO or N1 in diagnoses statistics database (130), which will result in the appropriately modified P when next computed. If only a small number N of data points have been collected for a given Boolean probability POi,Dnand they all haven the same value, then the probability of that value is 100%, which may be artificially high due to bias in a small sample. It may be more statistically appropriate to correct for this bias by using P* when computing degree of overlap DO for such data, as described in FIGs.3 A and 3B above and related text:(P - 0.5) p* > p > i VN

[0118] In the embodiment described in FIG. 2A, continuous observations associated with a given diagnosis are recorded in diagnoses statistics database (130) as number of data points Noi.Dn, a mean poi,Dn, and standard deviation ooi,Dn. For the first data value x, initialize N = 1, g = x, and o = 0. For continuous data, new value (x) is used to update these values as follows:Nnew—Nold + 1

[0119] The above equations are adapted from one of several mathematical approaches to incrementally computing means and standard deviations as taught in various textbooks such as Knuth, D. (1981) Semi Numerical Algorithms, Addison-Wesley publishers. The above new values may be entered into diagnoses statistics database (130). Note that the only diagnoses that are considered by Bayesian methods are diagnoses for which there is at least one instance in the database, so N=0 does not occur. When the first clinical record is encountered with the first instance of a particular observation O in a patient with diagnosis D, then N=1 and computation of G will result in a divide-by-zero error, so that condition must be trapped by the software. Furthermore, o2= o = 0 is nonsensical and may trigger other divide- by-zero errors in the Bayesian inference equation. In general, both the standard deviation Gand the variance G2may be biased artificially low when only one or a few samples are available, so it may be more statistically appropriate and mathematically expeditious to correct for this bias by using G* when computing degree of overlap for such data:

[0120] where oaii is the standard deviation for that observation from all patients in the database. Variance G2may then be computed from o*. For the very first value of this observation in any patient, oaii is itself zero, so the calculation of the probability density function PDF may generate a divide-by-zero error. For this case, we may force the value of P to be 0.05 for the one diagnosis with only one observation value. After that, the equation works well for N>2.

[0121] For continuous data related to time (e.g. age, duration, frequency), the internal representation of the data in diagnoses statistics database (130) may be computed from the logio of the data value in days, advantageously to reflect the tendency of time to be demarcated in such approximately logarithmic units. For example, values of duration may be recorded as log(day) = 0, log(hour) = -1.4, log(week) = 0.8, log(month) = 1.5, log(year) = 2.6). Values of frequency may be recorded as logio of the interval implied by the frequency, so “4 times a day” may be recorded as l / log(l / 4) = -1.7, while “4 times a week” may be recorded as l / log(7 / 4) = 4.1. If the clinician is allowed to specify such time values only as discrete categories (e.g. a symptom is described as present for months or years), the standard deviations computed from a number of such observations may be spuriously low (e.g. all patients with a particular diagnosis happen to have onset of months but actually varying over the range 1-11 months while all patients with a different diagnosis happen to have onset of years but actually varying from 1-9 years). This can cause the Bayesian inference to be very brittle, switching abruptly between probabilities of diagnoses. Instead, it is better to allow the clinician the ability to enter any number for any unit of time (e.g. hours, days, weeks, months, years) and then to convert that to days for the log transformation that is entered into EHR (112). This will result in a natural distribution of values that reflects the true variability of the presentation as well as the variability in the way that clinicians happen to ask questions and convert answers to such entries. If diagnoses statistics database (130) has been constructed with discretized values such as “present for months” and “present for years”, the standard deviations ooi,Dn may be modified so that they have a minimum value of approximately halfthe difference between the values of the two steps adjacent to the mean poi.Dn (0.55 in the example above for months and years in logio(days)).

[0122] Diagnoses that occur rarely in patient populations are likely to have only small numbers of observations associated with them and contributing to the distributions appearing in the cells of the diagnoses statistics database (130). Before observations are entered into the EHR (112) of a patient who has such a diagnosis, the initial probability of that diagnosis P(Dn) will be very low. Under normal circumstances, the common diagnoses in differential diagnosis list (142) with initially high probability will decrease in Bayesian probability as observations inconsistent with those common diagnoses accrue. Because the probabilities of all possible diagnoses must add to one, some of the rare diagnoses with initially low probability will necessarily increase in their probability, rising in differential diagnosis list (142) presented to physician (110). If a patient actually has a diagnosis that is not present in the database, then the probabilities of the incorrect diagnoses cannot decrease substantially even though the congruence between them and the accrued data in EHR (112) is not high. Phenotype completion tool (180) described above may be used to identify conditions in which a suggested diagnosis is thus artificially high. The phenotype for that diagnosis will include items that have a low congruence with the diagnosis, so the problem may be brought to the attention of the physician such as by using a distinctive color code in display device (160).

[0123] For each diagnosis in the database, a list of rule-out diagnoses with their codes may be assembled by experts. If the probability P(Dn) for all items in differential diagnosis list (142) is below some threshold, display device (160) may present to physician (110) a list of rule-out diagnoses for those items in differential diagnosis list (142) that have the highest probability P(Dn). This may remind a physician of a diagnosis that has been overlooked by the limited scope of diagnoses statistics database (130) and that is deserving of further research to identify diagnostic tests or other observations that Bayesian Exploration misses due to the limited scope of the database. This feature will be particularly important during initial implementation of system (100) when diagnoses statistics database (130) does not yet include many rare diagnoses.

[0124] The only diagnoses that system (100) will include in differential diagnosis list (142) are those for which at least one case exists in diagnoses statistics database (130). Any statistically meaningful Bayesian computation of posterior probability of a given diagnosis requires at least 3-5 instances of such diagnoses to estimate the probability of eachobservation associated with the phenotype of that diagnosis. The National Organization of Rare Disorders defines rare as a condition that affects fewer than 200,000 Americans or one out of 1667 people. There are estimated to be 10,000 such rare disorders. In the early stages of compiling diagnoses statistics database (130) when there may be fewer than 10,000 total patient records, most rare disorders will not have sufficient numbers of cases to compute their probability and many will not be represented at all. Nevertheless, it is important that, when appropriate, rare disorders at least appear in differential diagnosis list (142) for the physician’s consideration, particularly when the more common disorders that might have been responsible for the presenting clinical problem all have low probability as described above.

[0125] One way to overcome this problem is to deliberately add EHRs (112) that represent a given rare diagnosis, either by soliciting such records from other databases or by creating records that reflect its clinical phenotype as described in the literature. In either case, the initial probability of this rare diagnosis would then be over-estimated by simply counting the percentage of such cases in diagnoses statistics database (130). It is thus advantageous to provide a flag or identifier of such artificially added records so that this statistical distortion may be at least partially mitigated. Any EHRs (112) with such a diagnosis that carry this flag may have their contribution to the probability of the diagnosis appropriately discounted. When such flagged EHRs are added to the database, their observations may be integrated into the values in the appropriate cells of diagnoses statistics database (130) but they should not increment the counts of their diagnoses that may be used to determine the initial probabilities of those diagnoses. For example, suppose that diagnoses statistics database (130) has been constructed from 1000 unflagged EHRs (112) from real patients and 5 flagged EHRs constructed to represent a rare diagnosis. The probability of the rare diagnosis P(Di) would then be 1 / 1000 = 0.001, essentially an upper bound to its true incidence, instead of 5 / 1000 = 0.005 if the flagged records were counted. If the database expanded by 2000 unflagged records without a naturally occurring case of the rare diagnosis, the probability of that rare diagnosis would drop to 0.0005. If a naturally occurring case of the rare diagnosis were added among the new unflagged records, the probability bound would be (l+l) / 2000 = 0.001. Thus, any over-estimation will decrease asymptotically to the true probability as the database expands. As the number of naturally occurring cases of rare diagnoses increases in the database and becomes sufficient to compute their phenotypes, it may be advantageous to remove some or all data from the flagged EHRs from diagnoses statistics database (130).

[0126] One application of system (100) is to use the cumulative experience represented in diagnoses statistics database (130) to make recommendations about the anticipated effectiveness and / or cost / effectiveness of various treatments. This may be done by adding diagnoses reflecting the resolved or remission state or symptomatic improvement of each final diagnosis and computing the utility of treatments toward reaching those states according to Bayesian Exploration. In clinical medicine, this is complicated by the likelihood that patients will be prescribed more than one treatment (e.g. drugs + diet change + physical therapy) which vary in their individual anticipated utility, and that patients will not adhere completely to some or all prescriptions. During follow-up visits, it may be useful to obtain information about both adherence and response to treatment. It may also be useful to compute both a singular value for the degree of improvement in response to treatment and a singular value for the degree of adherence to the complete treatment regimen, as described below.

[0127] The degree of improvement may reflect differential effects on various symptoms associated with the diagnosis of which the patient is aware (e.g. pain, strength, etc.) and / or observations that the physician may make (e.g. histology, x-ray, serology, etc.). Each diagnosis may be accompanied by a list of treatment options, which may be predetermined by clinical experts and / or compiled continuously from the treatments actually prescribed for patients with that diagnosis as recorded in diagnoses statistics database (130). These treatments may differ in their intended effect on the various symptoms and observations associated with the diagnosis. The clinician may identify a relative weighting of the symptoms and observations that should be quantified in follow-up visits to assess an overall response to treatment.

[0128] The degrees of adherence to the various treatments may each be quantified by asking the patients what percentage of each prescribed treatment was actually received. These values may be weighted before combination according to their individual relative utility in achieving the overall response to treatment. The relative utility of each prescribed treatment may be defined by the clinician in consideration of the desired response to treatment or it may be extracted from diagnoses statistics database (130) by determining a correlation value between the prescribed treatment and the response to treatment as recorded for other patients with the same diagnosis and treatment plan. Such a correlation may take into consideration the degree of adherence of those other patients to their treatment and the similarity in the weighting of symptoms and observations used to assess their responses to treatment, using methods of multivariate analysis that are familiar to statisticians.

[0129] It may be valuable for physician (110) to consider the cost of treatments or observations when making selections. Cost-effectiveness may utilize a singular value for cost that reflects both the financial cost of the treatment or observation itself and non-monetary costs associated with obtaining the treatment or observation (e.g. time lost from work) and side-effects (e.g. reduction in quality of life, commonly quantified as Quality Adjusted Life Years [QA Y] and associated with a monetary value per QALY for reimbursement decisions). These various types of cost may be converted into a monetary equivalent and combined into a single cost of treatment or observation. The cost-effectiveness of an observation has been described in the prior art. The cost-effectiveness of a treatment may be assessed by its probability of contributing to the degrees of improvements recorded in the diagnoses statistics database (130) as extracted by Bayesian inference and a QALY analysis of the value of the improvements attributable to the treatment.

[0130] The digitization of EHRs has been underway for over 50 years, with many competing schemes, each of which has failings that frustrate those who attempt large scale analysis of such records. Most of these health record schemes were designed without knowledge of how they might be used, posing an insurmountable challenge for their designers. The structured, objective format of EHRs (112) is well-suited to the Bayesian inference engine (140) and Bayesian Exploration engine (150) of our system (100), but it has undergone substantial evolution of its details to correct for instances of poor performance. Our experience suggests that such iteration will continue and that it is an important aspect of any design for an EHR that it supports such continuing refinement.

[0131] Each time that a clinician selects an observation or diagnosis that has not been suggested by Bayesian Inference engine (140) constitutes an opportunity to improve the design and / or contents of EHR (112) and / or diagnoses statistics database (130). In our experience, such over-ride events sometimes reflect the sparsity of entries to diagnoses statistics database (130) for rare diagnoses or unusual presentations, which should be selflimited as the size and scope of the database naturally grows with the addition of new EHRs containing more observations and diagnoses. Some over-ride events, however, arise from ambiguities or omissions of observations that are critical for certain diagnoses. The selection and description of the set of possible observations in EHR (112) starts with the judgment of clinical experts who create the first edition of system (100), tempered by the need to avoid overwhelming the physician and the Bayesian engines with too many, perhaps overlapping observations.

[0132] The initial development of system (100) may start with experts examining textbooks and published case series to find disease defining signs and symptoms to include as Boolean type observations Oi. The initial instance of diagnoses statistics database (130) may be compiled from EHRs (112) that have been hand-curated from more conventional clinical records such as narrative notes and paper charts. Diagnoses statistics database (130) may then be queried to identify those Boolean observations that appear to be useful because they have some minimal positive value that is much greater than 0.5 = 50% probability P( Oi | Dn), or are consistently negative (P( Oi | Dn) « 0.5) in patients with a particular diagnosis. A list of such apparently useful observations may be used to supplement, validate, replace and / or refine the wording of possible observations in the EHRs. Expert analysis of those diagnoses that Bayesian Inference engine (140) omits or under-values in differential diagnosis list (142) provides actionable insights into observations whose descriptions need to be refined or added to improve performance. That effort may be efficiently triggered and targeted by reviewing the performance of system (100) and the contents of its diagnoses statistics database (130).

[0133] Each EHR (112) will generally consist of a relatively small number of observations out of a much larger number of possible observations to which physician (110) may or may not navigate as described above. EHR (112) may include places for all possible observations; those not made may be designated as empty. Alternatively, EHR (112) may include only the observations for which data have been explicitly entered, in which case it may be assumed that any other observations have not been made. The complete set of possible observations coded into system (100) may advantageously evolve as experience with the system accrues. New types of observations may become possible as new diagnostic technologies are developed. Boolean type observations may be reworded to make their salience more apparent to the physician. Some Boolean observations may be deemed useful or not useful in their negative or positive values, in which case it may be advantageous to include a flag with those observations that tells Bayesian Inference engine (140) which observation values to consider and which to ignore when recomputing probabilities of diagnoses Dn. For example, the description of a Boolean observation might be worded as “audible clicking in the joint” or as “no audible clicking in the joint”, so the same symptom might be coded as a 1 or 0 value.

[0134] Newly added observations may be at least partially redundant with previous observations and the new observations will initially have very few entries in the database. Nevertheless, each observation represents an added dimension of the clinical hyperspace forwhich observation utility matrices (137) must be computed across all items in the set of possible diagnoses. This represents a substantial computational load that may advantageously be performed off-line at regular intervals or as warranted according to how many new EHRs have been added. During real-time computation of Bayesian Exploration, each observation utility matrix (137) must be weighted by the Bayesian prior probabilities of all diagnoses and its weighted elements summed to identify the total probability -weighted utility TPUoi for each diagnosis Oi that may be displayed in observation list (152). This represents a substantial computational load that is performed frequently and ideally returns results with a time lag that is short enough to be negligible to the physician during the clinical encounter. Thus, it may be advantageous to exclude observations that are unlikely to add any discriminative utility so as to reduce these computational burdens.

[0135] A new observation becomes useful only when there are enough values of that observation in association with a given diagnosis for the statistical spread of those values to be reasonably small. The methods described above for correcting standard deviations to deal with small numbers of available continuous data values or correcting probabilities for small numbers of Boolean data values result in large statistical spreads until the square root of the number of values available for that observation is sufficiently large as to reduce the default standard deviation or probability, respectively. Furthermore, the observation is only useful when at least two diagnoses have a sufficient number of observations such that those diagnoses may be differentiated based on the utility of that observation Uoi,Dm,n. It may be advantageous to search the set of all observations to identify those that do not yet meet some threshold criteria for number of values associated with at least two diagnoses and flag them for exclusion from the Bayesian analyses. If and when the addition of new EHRs into diagnoses statistics database (130) crosses the threshold criteria, the exclusionary flag may be automatically removed. An observation deemed wholly or partially obsolete may be removed from the set of observations available to physicians. It may be deemed useful to flag that observation for exclusion from further Bayesian analyses upon the judgment of an expert or upon a replacement observation crossing a threshold for number of values of such replacement observations in the database. Means may be provided in the list of observations to flag those observations that are so excluded, thereby reducing the computational loads.COMPUTING SYSTEM

[0136] FIGs. 2B, 2C, 3B, 3C, 4B, 5, and 6 show flowcharts describing example methods according to embodiments of the disclosure. One or more of the steps in these figures may be performed by system (100), e.g., by one or more processors of a computing system as described in reference to FIG. 8.

[0137] While the various steps in the flowcharts are presented and described sequentially, one of ordinary skill will appreciate that some or all of the steps may be executed in different orders, may be combined or omitted, and some or all of the steps may be executed in parallel. Additional steps may further be performed. Furthermore, the steps may be performed actively or passively. For example, some steps may be performed using polling or be interrupt driven in accordance with one or more embodiments of the disclosure.

[0138] Embodiments of the technology may be implemented on a computing system. Any combination of mobile, desktop, server, embedded, or other types of hardware may be used. For example, as shown in FIG. 8, the computing system (800) may include one or more computer processor(s) (802), associated memory (804) (e.g., random access memory (RAM), cache memory, flash memory, etc.), one or more storage device(s) (806) (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory stick, etc.), and numerous other elements and functionalities. The computer processor(s) (802) may be an integrated circuit for processing instructions. For example, the computer processor(s) may be one or more cores, or micro-cores of a processor. The computing system (800) may also include one or more input device(s) (810), such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. Further, the computing system (800) may include one or more output device(s) (808), such as a screen (e.g., a liquid crystal display (LCD), a plasma display, touchscreen, cathode ray tube (CRT) monitor, projector, or other display device), a printer, external storage, or any other output device. One or more of the output device(s) may be the same or different from the input device(s). The computing system (800) may be connected to a network (812) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) via a network interface connection (not shown). The input and output device(s) may be locally or remotely (e.g., via the network (812)) connected to the computer processor(s) (802), memory (804), and storage device(s) (806). Many different types of computing systems exist, and the aforementioned input and output device(s) may take other forms.

[0139] Software instructions in the form of computer readable program code to perform embodiments of the technology may be stored, in whole or in part, temporarily or permanently, on a non -transitory computer readable medium such as a CD, DVD, storage device, a diskette, a tape, flash memory, physical memory, or any other computer readable storage medium. Specifically, the software instructions may correspond to computer readable program code, that when executed by a processor(s), is configured to perform embodiments of the technology.

[0140] Further, one or more elements of the aforementioned computing system (800) may be located at a remote location and connected to the other elements over a network (812). Further, embodiments of the technology may be implemented on a distributed system having a plurality of nodes, where each portion of the technology may be located on a different node within the distributed system. In one embodiment of the technology, the node corresponds to a distinct computing device. Alternatively, the node may correspond to a computer processor with associated physical memory. The node may alternatively correspond to a computer processor or micro-core of a computer processor with shared memory and / or resources.

[0141] While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as disclosed herein. Accordingly, the scope of the invention should be limited only by the attached claims.

[0142] Although a limited number of example embodiments have been described in detail above, those skilled in the art will readily appreciate that many modifications are possible in the example embodiments without materially departing from this disclosure. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the following claims.

Claims

CLAIMSWhat is claimed is:

1. A method for dynamically adapting an anticipatory user interface of an electronic health record system, the method comprising: determining, by the electronic health record system and using Bayesian inference, for a patient, a first differential diagnosis list comprising a first subset of diagnoses selected from a plurality of potential diagnoses; determining, by the electronic health record system, for the patient, a first observation list comprising a first subset of observations selected from a plurality of potential observations, the first subset of observations selected based on having a largest utility for disambiguation of the first differential diagnosis list; dynamically adapting the anticipatory user interface to operate in a first mode, based on the first differential diagnosis list and the first observation list, wherein the operating of the anticipatory user interface in the first mode comprises: displaying, in the anticipatory user interface, the first differential diagnosis list to a physician; displaying, in the anticipatory user interface, for each diagnosis of the first subset of diagnoses, a current probability that the diagnosis accounts for previously made observations upon the patient; displaying, in the anticipatory user interface, the first observation list to the physician; displaying, in the anticipatory user interface, for each observation on the first observation list, a current utility of the observation to distinguish among the first subset of diagnoses in the first differential diagnosis list; after displaying the first observation list to the physician, receiving, by the anticipatory user interface, a first ambiguous input from the physician; resolving the first ambiguous input by probabilistically determining an association of the first ambiguous input with at least one particular observation of the first observation list; andobtaining at least one value for the at least one particular observation and enter the at least one value into an electronic health record of the patient.

2. The method of claim 1 further comprising, after entering the at least one value into the electronic health record of the patient: determining, by the electronic health record system and using Bayesian inference, for the patient, a second differential diagnosis list comprising a second subset of diagnoses selected from the plurality of potential diagnoses; determining, by the electronic health record system, for the patient, a second observation list comprising a second subset of observations selected from the plurality of potential observations, the second subset of observations selected based on having a largest utility for disambiguation of the second differential diagnosis list; dynamically adapting the anticipatory user interface to operate in a second mode, based on the second differential diagnosis list and the second observation list, wherein the operating of the anticipatory user interface in the second mode comprises: displaying, in the anticipatory user interface, the second differential diagnosis list to a physician; displaying, in the anticipatory user interface, for each diagnosis of the second subset of diagnoses, a current probability that the diagnosis accounts for previously made observations made upon the patient, including observations made while the anticipatory user interface was operating in the first mode; displaying, in the anticipatory user interface, the second observation list to the physician; displaying, in the anticipatory user interface, for each observation on the second observation list, a current utility of the observation to distinguish among the second subset of diagnoses in the first differential diagnosis list;after displaying the second observation list to the physician, receiving, by the anticipatory user interface, a second ambiguous input from the physician; and resolving the second ambiguous input by probabilistically determining an association of the second ambiguous input with at least one particular observation of the second observation list or a particular diagnosis of the second differential diagnosis list.

3. The method of claim 2, further comprising, while operating the anticipatory user interface in the second mode: displaying, to the physician, a list of phenotypical observations for the particular diagnosis; receiving, from the physician, at least one input indicative of a value of at least one phenotypical observation; and entering the value of the at least one phenotypical observation into the electronic health record.

4. The method of claim 2, further comprising, while operating the anticipatory user interface in the second mode: discounting at least one diagnosis of the second subset of diagnoses by setting the current probability of the discounted diagnosis to zero; and recomputing the current probabilities of other diagnoses of the second subset of diagnoses after setting the current probability of the discounted diagnosis to zero.

5. The method of claim 2, further comprising, while operating the anticipatory user interface in the second mode: displaying a list of treatment options for the particular diagnosis of the second differential diagnosis list; and receiving an input from the physician that selects one or more of the treatment options.

6. The method of claim 5, wherein to display the second observation list to the physician, the anticipatory user interface is configured to include observations of at least oneselected from a group consisting of an adherence and a response to a selected treatment option.

7. The method of claim 1, wherein the first ambiguous input from the physician comprises speech recorded by a microphone and processed by a natural language processor, and wherein to resolve the first ambiguous input, the anticipatory user interface is configured to: multiply a confidence of an output of the natural language processor times the utilities of the observations on the first observation list to obtain one product for each observation on the first observation list; and select the observation with the highest product as the at least one particular observation.

8. The method of claim 1 wherein displaying, in the anticipatory user interface, the first observation list comprises displaying the first observation list as one or more sublists in which the observations in each sublist may be obtained by one type of interaction with the patient.

9. The method of claim 1 wherein the current utility value displayed for each observation in the observation list considers an expected cost of the observation.

10. The method of claim 1, wherein the at least one value of the at least one particular observation is obtained from a source external to the electronic health record system, and wherein to obtain the at least one value, the anticipatory user interface is configured to: send a request to the external source for the at least one particular observation to be performed; and receive from the external source the at least one value.

11. The method of claim 10 wherein the at least one value is entered automatically into the electronic health record of the patient by the electronic health record system.

12. The method of claim 1, further comprising automatically generating a narrative note from information in the electronic health record of the patient.

13. The method of claim 1, further comprising adding information in the electronic health record of the patient to a diagnoses statistics database that contains probabilities that observations have specific values in patients with specific diagnoses, wherein these probabilities are used to determine the first differential diagnosis lists and the first observation list for future patients.

14. The method of claim 13, further comprising correcting the probabilities used to determine the first differential diagnosis list and the first observation list for a total number of values contributing to those probabilities.

15. The method of claim 13, further comprising computing and saving, in the diagnoses statistics database, a utility of each observation for distinguishing various pairs of diagnoses in the diagnoses statistics database.

16. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions executed by one or more processors associated with an electronic health record system, the plurality of machine-readable instructions causing the one or more processors to perform the method of any of claims 1 to 15.

Images