Automated parallel processing of decision tree guidelines using electronic records
The method transforms structured data into multidimensional spaces using decision trees to efficiently select protocols, addressing inefficiencies in protocol selection by automating navigation and handling variable data.
Patent Information
- Application Number
- JP2024532959
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-03
- Filing Date
- 2022-11-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-11-23
AI Technical Summary
Selecting effective protocols is challenging due to variable data types and sizes, frequent guideline changes, and differing decision-maker interpretations, leading to inefficiencies and complexity in protocol selection.
A computer-implemented method transforms structured data sets into multidimensional spaces using decision trees, identifying eigenpoints and similarity scores to efficiently navigate and update protocols, handling missing values and reducing iterative processing.
Facilitates consistent and efficient protocol selection by automating the navigation of complex decision trees, saving time and resources while adapting to changes in guidelines.
Smart Images

Figure 2024542775000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 285,685, filed December 3, 2021, which is incorporated by reference herein in its entirety for all purposes. [Background technology]
[0002] Protocols are frequently used to address specific problems or to try to avoid or postpone subsequent undesirable consequences. However, selecting a protocol for a particular use case can be difficult because a protocol's likelihood of being effective often depends on current conditions and event history.
[0003] For example, assume that a protocol is used to protect a computing system from security threats. The effectiveness of a given protocol may depend on the type of data stored in the computing system, the type of network to which the computing system is connected, whether the computing system has been previously accessed without authorization, etc. As another example, assume that a protocol is used to inhibit disease progression in a particular human subject. The effectiveness of a given protocol may depend on which disease subtype the subject has, the current level of disease progression, and the demographics of the subject.
[0004] Moreover, selecting a protocol often involves combining and considering different combinations of conditions and different combinations of antecedent events, after which different decision makers may arrive at different protocol choices.
[0005] Furthermore, different types and amounts of data may be available at different times to inform the choice of protocol, and therefore it may be particularly difficult to identify techniques that can reliably convert input data of variable size and type into useful output.
[0006] Guidelines are used in several industries to remove noise and variability across protocol selection and facilitate consistent, interpretable protocol selection. However, some guidelines (e.g., clinical practice guidelines) are particularly complex (e.g., when printed, contain protocol elements that span hundreds of pages of cross-references) and / or change frequently as new information becomes available. Thus, it may be difficult for a human decision maker to navigate the guidelines efficiently, and the frequent changes may make them expensive to implement via software. For example, changes in upstream considerations or decisions may affect how some or all of the downstream decisions should be made. (Consider a case where considerations change from characterizing whether a computing system or subject's age is in one or two age groups to whether the age is in one or four age groups. All four age groups may then be associated with different subsequent considerations compared to those of the original two age groups.) Complexity and frequent changes may cause guidelines to be used less frequently.
[0007] It would therefore be advantageous to identify and use techniques that facilitate more efficient and consistent selection of protocols. Summary of the Invention
[0008] A computer-implemented method for predicting a state or an effective protocol using transformations and projections is provided. A structured data set including a first set of key-value pairs is accessed. Each of the first set of key-value pairs characterizes an evaluation outcome or a protocol characteristic of a subject. The first set of key-value pairs is transformed into a second set of key-value pairs, where at least some keys of the second set of key-value pairs are different from each key of the first set of key-value pairs. The second set of key-value pairs is projected to identify a subject unique point in a multidimensional space. One or more decision trees including a plurality of decision nodes and a plurality of leaf nodes are accessed. Each of the plurality of leaf nodes is connected to a root node via a leaf-node unique trajectory. Each of the plurality of decision nodes corresponds to a criterion based on at least one value in the second set of key-value pairs. For each leaf node in the one or more decision trees, a leaf-node unique point in the multidimensional space is determined based on the leaf-node unique trajectory. For each leaf node in the one or more decision trees, a similarity score is determined based on the leaf-node unique point and the subject unique point. Leaf nodes of the multiple incomplete subsets are identified based on the similarity scores. State or protocol information associated with each leaf node in the incomplete subset is obtained. Output associated with an object that includes the state or protocol information is generated.
[0009] For each leaf node in the one or more decision trees, determining the leaf-node-specific points may include converting the leaf-node-specific trajectories into a first leaf-node-specific data set using text extraction, converting the first leaf-node-specific data set into a leaf-node-specific set of key-value pairs, and projecting the second leaf-node-specific set of key-value pairs to identify the leaf-node-specific points.
[0010] Determining the similarity score may include applying a cosine similarity function.
[0011] The method may further include, for each term of the term set, determining a reverse trajectory frequency indicating the frequency at which the term occurs across leaf-node-specific trajectories associated with leaf nodes in the one or more decision trees, and / or determining a term frequency indicating the frequency at which the term occurs in each of the leaf-node-specific trajectories, wherein values of the second set of key-value pairs are determined based on the determined reverse trajectory frequencies and / or term frequencies.
[0012] The output may further include, for each node in the incomplete subset, a query represented by a decision node in the leaf-node-specific trajectory.
[0013] The output can include protocol information identifying potential treatments for the subject.
[0014] The structured data set may include, for each of the first set of key-value pairs, an initial timestamp, and converting the first set of key-value pairs to the second set of key-value pairs may include classifying a particular key-value pair of the first set of key-value pairs as an indexing event; for each key-value pair of the first set of key-value pairs, generating a modified timestamp using the initial timestamp associated with the key-value pair and the initial timestamp associated with the particular key-value pair; detecting that decision nodes in the one or more decision trees include a query to determine whether a particular event occurred within a particular time period relative to an occurrence of another particular event, where the other particular event corresponds to the indexing event; executing a query to determine whether the first set of key-value pairs represents the particular event and includes the first particular key-value pair associated with a modified timestamp within a particular time range; and determining a second particular key-value pair based on results of the query, where the second set of key-value pairs includes the second key-value pair.
[0015] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product including instructions configured to cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein.
[0016] The terms and expressions employed are used as terms of description and not of limitation, and the use of such terms and expressions is not intended to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the claimed invention. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it is to be understood that modifications and variations of the concepts disclosed herein may be made by those skilled in the art, and such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims. [Brief description of the drawings]
[0017] The present disclosure is described in conjunction with the accompanying drawings, in which:
[0018] [Figure 1] 1 illustrates a computing network 100 for transforming decision trees to facilitate more efficient processing and updating and using the transformed decision trees in accordance with some embodiments of the present invention.
[0019] [Diagram 2]1 illustrates an exemplary process for transforming decision trees and datasets by generating and using multi-dimensional spaces to identify predicted state or protocol information, according to some embodiments of the present invention.
[0020] [Figure 3A] 1 illustrates a selected portion of an exemplary decision tree. [Figure 3B] 1 illustrates a selected portion of an exemplary decision tree.
[0021] [Figure 4] 4 shows a corresponding portion of an exemplary unstructured dataset of interest.
[0022] [Figure 5A] FIG. 1 illustrates an exemplary juxtaposition between variables corresponding to data available in exemplary target data (e.g., used to generate structured target data) and variables related to queries identified within decision nodes in an exemplary decision tree. [Figure 5B] FIG. 1 illustrates an exemplary juxtaposition between variables corresponding to data available in exemplary target data (e.g., used to generate structured target data) and variables related to queries identified within decision nodes in an exemplary decision tree. [Figure 5C] FIG. 1 illustrates an exemplary juxtaposition between variables corresponding to data available in exemplary target data (e.g., used to generate structured target data) and variables related to queries identified within decision nodes in an exemplary decision tree.
[0023] [Figure 6A] This shows a node in a decision tree that references (links) two principle pages.
[0024] [Figure 6B] 1 shows information corresponding to a structured dataset and mapped information regarding navigation transformed into a decision tree.
[0025] [Figure 6C] 1 illustrates criteria in two exemplary decision tree paths.
[0026] [Figure 7] 13 illustrates an exemplary performance metric that characterizes the accuracy of results produced using projections in a multi-dimensional space.
[0027] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes the similar components. If only a first reference label is used in this specification, the description is applicable to any one of the similar components having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0028] I. Overview In some embodiments, the decision tree of the set of guidelines is transformed by detecting terms that distinguish different trajectories in the decision tree. Each trajectory and / or leaf node in the decision tree can then be represented as a point in a multi-dimensional space, where different dimensions correspond to different terms (e.g., which may include key-value pairs, keys from key-value pairs, conditions, or potential outcomes of evaluating one or more conditions), and the value of each dimension indicates the frequency with which the term is present in the corresponding trajectory. A use-case-specific data set (e.g., corresponding to an individual subject or system) can include a set of key-value pairs, which can be transformed to map to use-case-specific points in the multi-dimensional space. Furthermore, each leaf node of the decision tree (which may include nodes that represent potential outcomes of a decision, but do not necessarily represent the end of a trajectory) can be projected into the multi-dimensional space. The distance between the use-case-specific point and each point associated with the leaf node can be determined, and the use case can be assigned to the leaf node associated with the shortest distance. Information corresponding to the leaf node can then be obtained and output.
[0029] This approach therefore avoids repetitive and time-consuming processing of queries in the decision tree. The transformation technique further facilitates handling of datasets with missing values. Instead of being "stuck" at a decision node due to missing values, the applicable corresponding queries can essentially be ignored or considered in the alternative. Furthermore, by avoiding an iterative approach to navigating the decision tree, the code defined to identify the leaf nodes for a given use case can be easily updated in response to detecting changes in the decision tree without the need to unwrap a series of nested conditional statements. II. An Exemplary System for Parallel Processing of Decision Trees
[0030] 1 illustrates an exemplary network 100 for transforming decision trees and using the transformed decision trees to facilitate more efficient processing and updating. Network 100 includes a decision tree processing system 105, which may include a computing system (e.g., a cloud computing system, a server, one or more computers, etc.). Decision tree processing system 105 includes a decision tree monitor 110 that accesses one or more decision trees 115. Decision tree monitor 110 may access decision trees 110 by (for example) scraping data from a web page that includes decision trees 110, downloading a file that includes decision trees, or receiving an electronic message (including decision trees 115) from another computing system.
[0031] The decision tree 115 includes a set of decision nodes and a set of leaf nodes. Each decision node includes a query. Each leaf node represents a particular protocol or a particular predicted state. For example, a leaf node may represent a predicted current security risk in a computing system or a predicted state of a medical condition of a subject. As another example, a leaf node may represent a protocol including one or more specific actions that may be performed to facilitate securing a computing system or treating a condition.
[0032] Each decision node query can request values of variables relevant to a prediction or recommendation in the guideline (e.g., notification). Each decision node query can include an evaluation of one or more terms and / or one or more subject-specific key-value pairs (e.g., each of which can correspond to a given term and a particular but associated value). The evaluation can include determining whether each, which, or any of the one or more criteria are met.
[0033] In some cases, leaf nodes in a given decision tree 115 are unique with respect to each other, but multiple trajectories may connect the root node to a given leaf node. For example, each leaf node may be associated with information identifying a different recommended protocol (e.g., treatment), but a given leaf node may be connected to the root node via multiple sets of potential query responses. In some cases, each leaf node in a decision tree is connected to the root node via only one trajectory, such that each leaf node corresponds to a particular set of query responses. In this case, at least some leaf nodes may be associated with the same information (e.g., may correspond to the same protocol recommendation).
[0034] The text extractor 120 extracts text from the decision tree 115 to detect queries in the decision nodes. For each unique leaf node, the text extractor 120 defines a leaf-node-specific dataset 125 to include one or more text strings that identify queries and responses associated with the trajectories connecting the root node and the leaf node.
[0035] Some words (or strings of numbers or other characters) may contribute to distinguishing between different leaf-node-specific datasets 125, while other words may not. For example, all leaf-node-specific datasets 125 may contain many uses of the word "the," meaning that detecting the word "the" does not provide information about which leaf-node-specific datasets 125 are being evaluated. On the other hand, detecting a string of words indicating that a particular fourth-line treatment has previously been administered may provide substantial information about which leaf-node-specific datasets 125 are being evaluated.
[0036] Thus, the tokenization controller 130 can collectively process the leaf node-specific datasets 125 corresponding to multiple leaf nodes in the decision tree to identify the extent to which various terms distinguish the datasets. The process can include determining, for each term and each leaf node-specific dataset, the number of times the term occurred in the dataset. This amount can be (but need not be) normalized based on the total number of terms in the leaf node-specific datasets to define the term frequency. A term can include a single word, multiple words, a text string (including one or more spaces or lacking one or more spaces), etc. The tokenization controller 130 can be configured to break down long text into words and phrases. The process can further include determining, for each term, an inverse document frequency based on the number of different leaf node-specific datasets in which the term occurred. The inverse document frequency can be defined as the logarithm of the total number of leaf node-specific datasets divided by the number of leaf node-specific datasets in which the term occurred. The vectorization controller (not shown) can generate a provisional score for each term-trajectory pair based on the term frequency and the inverse document frequency. For example, the interim score may be the product of the term frequency and the inverse document frequency in the leaf node-specific dataset 125. A score may be further defined for each term (across trajectories) as (for example) the maximum of the interim scores associated with the term. In some cases, the vectorization controller (not shown) uses a term frequency-inverse document frequency technique to assign a term score to each term.
[0037] In some cases, supervised learning is used to facilitate the identification of terms that support predictions regarding which leaf nodes a trajectory will connect from a decision node. For example, a set of records can be labeled (e.g., by a human or by using code) to indicate that evaluation of a decision node based on the records results in a movement from the decision node to a given next node. To illustrate, the set of records can include a report characterizing one or more CT scans, and the label can indicate a partial or complete trajectory of a decision tree (e.g., terminating at a node corresponding to a recommended subsequent action) whether or not the number of tumors exceeded a predetermined threshold. The label can be identified manually by a human reviewer or assisted by a computer. Various techniques (e.g., supervised learning) can be used to iteratively improve the label prediction. The label can indicate which stage of cancer a given current progression corresponds to (e.g., when various branches from a decision node correspond to different stages of progression) or whether the report indicates the extent to which a given disease has progressed in a particular period of time. A machine learning model (e.g., a natural language model) can then use the various reports and labels to identify one or more selection features, tokens, and / or key-value pairs that are informative as to how to evaluate the decision of the decision node.
[0038] When decision trees are particularly large, it can be difficult to obtain enough labels to support fully supervised learning. This data problem is amplified by the fact that the trajectory directions corresponding to individual decision nodes may be imbalanced. To illustrate, a given decision node may correspond to 99.99% of the instances going in one direction, compared to 0.01% going in another direction. To ensure that the model is not biased, it may be important to ensure that there are enough samples corresponding to the other directions and / or to apply balancing techniques.
[0039] The extent to which this is possible and / or the cost (e.g., time, resources, and / or financial costs) of this approach can be complex, as there may be multiple unbalanced decision nodes across layers within a decision node. Thus, it may be advantageous to focus the labeling effort. The focus may include prioritizing (for example) under-represented trajectory labels and / or prioritizing one or more under-represented features. In some cases, the focus may include prioritizing one or more (for example) under-represented features associated with one or more trajectories to result in under-represented intermediate nodes or under-represented edge nodes. For example, feature importance techniques may be applied to identify the degree to which various decision nodes influence the trajectory output (e.g., taking into account the underlying branch probabilities and / or data availability) and / or previously observed or predicted imbalances for decision nodes. Thus, a prioritization may be set that predicts the importance of reserving context-specific labels corresponding to unrepresented path labels.
[0040] In some cases, tokens can be identified based on learned features that are important to the progress of the decision node. In some cases, training may be performed (for example) that utilizes tokens that correspond to context-specific labels (e.g., and potentially other data). These predictions can be used to predict values that were missing from the input dataset and / or to predict partial or complete remaining portions of the trajectory that correspond to a given instance.
[0041] The tokenization controller 130 may define a key for each term that is assigned a term score that exceeds or meets a predefined threshold (e.g., an absolute or relative threshold). For each leaf node, the tokenization controller 130 may then define a set of leaf node-specific key-value pairs 135 that include or represent each defined key and that include (for each key) a corresponding value that indicates whether the key was detected in the corresponding leaf node-specific data set. It will be understood that the "value" corresponding to a key or token may include (for example) a particular category, a particular numeric value, a value within a particular range, a category within a particular group of categories, or a value that satisfies a given condition (which may potentially further depend on a value corresponding to another key or variable). The tokenization controller 130 breaks each text string in the decision tree path (or of the subject data) into tokens (e.g., n-grams that may include one word, two words, etc.). The values of the key-value pairs may be defined to identify the number or frequency (across multiple terms) that the key was detected in the text associated with the leaf node. It will be understood that a key may correspond to any of one or more words, numbers, symbols, etc. For example, a key-value pair may indicate that the number of lesions detected in a scan of a subject collected within the last week is 70%-90% of the number of lesions detected in a scan obtained by imaging the same subject 2-3 weeks prior. Thus (for example), tokens may be identified based on supervised or unsupervised learning, where the tokens may identify one or more particular variables and one or more conditions (e.g., including one or more thresholds or matches) to which the tokens pertain. The key may identify that a variable of interest at a given decision node is related to the relative number of lesions, and the value may identify the exact relative number of lesions (or a given range within which the exact relative number of lesions falls).
[0042] A vectorization controller (not shown) can store a set of values for each of the sets of leaf-node-specific key-value pairs 135 as leaf-node-specific points 145 in a multi-dimensional space, with one dimension representing different decision tree trajectories and another dimension representing different keys. The leaf-node-specific points 145 can be separate from each other in the multi-dimensional space because the keys may have been defined by prioritizing the selection of terms that were only in a subset of the leaf-node-specific data set 125. In some cases, the leaf-node-specific points 145 are defined to include scaled versions of the values of the sets of leaf-node-specific key-value pairs 135, with a scaling factor applied to each value based on the term score (e.g., positively correlated) associated with the corresponding key. In some cases, the leaf-node-specific points 145 can include at least one value that corresponds to or represents a range or category. In this case, the leaf-node-specific points 145 can correspond to a region, volume, or subspace in the multi-dimensional space.
[0043] The user device 150 can send a communication to the decision tree processing system 105 to request that one or more leaf nodes and / or information related to one or more leaf nodes be identified for a particular use case. For example, the request can correspond to a request to identify any leaf nodes associated with a current or recent time point for a subject, or any leaf nodes associated with a probability of occurrence above a predetermined threshold for a subject (e.g., during a given time period or at a given future time point). A particular use case can correspond to a particular subject at a particular time point. For example, where each leaf node is associated with information identifying a particular treatment protocol, the user device 150 can request identification of the top five treatment recommendations for a given subject at the current time.
[0044] The request may include a use case specific structured dataset 155 or may include information on how to access the use case specific structured dataset 155. The use case specific structured dataset 155 may include a set of key value pairs corresponding to a particular use case (e.g., a particular subject). The use case specific structured dataset 155 may include one or more log messages (e.g., may include key value pairs). The use case specific structured dataset 155 may include part or all of an electronic health record that includes time-stamped sequentially ordered key value pairs (e.g., in one or more log messages) that convey (for example) the subject's demographics, past diagnoses, any current diagnoses, recent symptoms, lab results, imaging results, vital signs, specialist assessments, hospitalizations, etc. For each of one, more, or all keys in the use case specific structured dataset 155, the key is potentially not present in the leaf node specific key value pairs 135.
[0045] The data in the use-case-specific structured data sets 155 may initially come from multiple sources. A data aggregator 160 may collect the data from these sources and structure the data (e.g., using predefined keys).
[0046] The interface controller 165 of the decision tree processing system 105 can receive requests from the user device 150 and the use case specific structured dataset 155. The tokenization controller 130 can then convert the use case specific structured dataset 155 into use case specific key value pairs 170 that contain the same keys as those used in the leaf node specific key value pairs 135. Because the use case specific structured dataset 155 can already contain key value pairs, a search for a specific term may be insufficient to detect whether a given dataset contains information about a given key defined using the decision tree 115. Rather, a mapping and / or lookup table may be used to determine, for each key defined by the tokenization controller 130, how to determine the corresponding value using the use case specific structured dataset 155. In some cases, the mapping may be a one-to-one mapping where the mapped keys are identical or represent the same or very similar type of data. In some cases, the mapping associates a single key defined by the tokenization controller 130 with multiple keys in the use case specific structured dataset 155. For example, the key “malignant lymphoma” may be mapped to ((Hodgkin's lymphoma: YES) OR ("Non-Hodgkin's lymphoma: YES) AND (malignant: YES)), thereby relating to three keys in the use case specific structured dataset 155. After the tokens are identified, to identify the mapping, the tokenization controller 130 may use a transformation lookup table to associate the key defined by the tokenization controller 130 with one or more keys in the use case specific structured dataset 155 and identify any logic and / or mathematics to be used to transform the corresponding values.
[0047] In some cases, the mapping involves evaluating timestamps in the use case specific structured dataset 155. Evaluating the timestamps can include defining an index date (e.g., as the date a particular diagnosis was made) and modifying other timestamps to correspond to the index date. Then, if a key is defined as representing whether a given event occurred during a period corresponding to the diagnosis date (e.g., a given treatment was taken or a given stage of a disease was reached), the tokenization controller 130 can determine whether the use case specific structured dataset 155 includes a log message that identifies a code corresponding to an event associated with a modified timestamp that is shorter than the duration of the period.
[0048] The projection controller 140 can store the set of values of the use-node-specific key-value pairs 170 as use-case-specific points 175 in the multi-dimensional space. In some cases, the values are scaled based on the distance between the use-case-specific points 175 and the leaf-node-specific points.
[0049] The similarity score may be based on the distance between the points and / or may be negatively correlated with the distance between the points, for example, the similarity score may be the inverse of the distance, the negative of the distance, or the distance minus a constant.
[0050] In some cases, the similarity scores and / or distances are generated using cosine similarity scores, correlation, or another comparison metric.
[0051] The similarity score controller 180 can use the similarity score to select a leaf node of an incomplete subset for the request. The incomplete subset can correspond to the leaf nodes associated with the highest similarity scores relative to other leaf nodes in the decision tree 115. The incomplete subset can be defined as leaf nodes that correspond to similarity scores that exceed a threshold absolute or relative similarity score. For example, the incomplete subset can include leaf nodes that correspond to the n highest similarity scores, where n is predefined or specified by a user.
[0052] The decision tree monitor 110 can retrieve information corresponding to each leaf node in the subset. The information can include (for example) the name of the leaf node, the leaf node's metadata, and / or content cited in the leaf node representation (e.g., content from the leaf node link file or document). The information can include a predicted current state (e.g., a predicted stage of a disease, or a predicted security level against a malicious threat) or a recommended protocol (e.g., a treatment plan or security enhancements) to implement. For example, the protocol can identify a composition or active agent dosage, and / or a schedule (indicating when the composition or active agent is to be administered).
[0053] The interface controller 165 can transmit the retrieved information for the subset of leaf nodes (or a processed version thereof) to the user device 150. The interface controller 165 can further identify key-value pairs associated with each of the subset of trajectories. Thus, the user can identify which key-value pairs resulted in a given predicted state or identification of a given protocol. The identification can identify leaf node specific key-value pairs and can also identify any deviations in such pairs in the use case specific key-value pairs 170. Thus, any trajectory deviations can be communicated even if the user's key-value pairs do not match each key-value pair of the trajectory. Furthermore, this automation facilitates non-iterative processing, which saves significant time over conventional techniques for traversing decision nodes and saves enormous time over a human attempting to go through a complex decision tree (e.g., resolving an unclear decision). III. An Exemplary System for Parallel Processing of Decision Trees
[0054] FIG. 2 illustrates an exemplary process for transforming decision trees and datasets by generating and using a multi-dimensional space to identify predicted state or protocol information according to some embodiments of the present invention.
[0055] In block 202, the interface controller 165 accesses a structured dataset that includes a first set of key-value pairs. The structured dataset can correspond to a particular structure (e.g., includes keys from a predefined set of keys, has key-value pairs present in a particular order, and / or is in a log file). The structured dataset can be a use case specific structured dataset.
[0056] At block 204, the tokenization controller 130 transforms the first set of key-value pairs into a second set of key-value pairs. Each of at least some of the keys of the second set of key-value pairs may be different from any of the keys of the first set of key-value pairs. The transformation may include (for example) determining a given second key-value pair of the second set of key-value pairs by identifying a single corresponding value in the first set of key-value pairs, performing a calculation using multiple values in the first set of key-value pairs, or performing a transformation of a single corresponding value of the first set of key-value pairs (e.g., mapping numeric values to ranges or vice versa, identifying negatives of values, converting units, etc.).
[0057] The keys of the second set of key-value pairs may be identified by the tokenization controller 130 to include variables that, alone or in combination, are beneficial in distinguishing different trajectories in one or more decision trees. The scores of the second set of key-value pairs may have been selected using TF-IDF to highlight terms that are expressed as being significantly more frequent in the leaf-node trajectories of the incomplete subset than others.
[0058] At block 206, the projection controller 140 projects the second set of key-value pairs to identify the target unique points in the multi-dimensional space. For example, the target unique points can include some or all of the values of the second set of key-value pairs. As another example, one, more, or all of the values in the second set of key-value pairs can each be scaled (e.g., based on the TF-IDF value associated with the corresponding key), and the target unique points can include the scaled values.
[0059] In block 208, the decision tree monitor 110 accesses one or more decision trees. Each of the one or more decision trees includes a set of decision nodes and a set of leaf nodes. The decision trees can correspond to one or more guidelines (e.g., regarding how to diagnose or treat a particular subject).
[0060] At block 210, the projection controller 140 determines a leaf-node-specific point in the multi-dimensional space for each leaf node in the set of leaf nodes. For example, text from the decision tree can be extracted, and a set of leaf-node-specific key-value pairs can be identified. The key-value pairs can include keys that are identified by the tokenization controller 130 to include variables that, alone or in combination, are beneficial in distinguishing different trajectories in one or more decision trees. The keys of the leaf-node-specific key-value pairs may have been selected using n-gram tokenization. TF-IDF can be utilized to calculate the values of the key-value pairs to indicate the extent to which a given term appears significantly more frequent than other terms. Some or all of the keys of the leaf-node-specific key-value pairs may be the same as or correspond to the keys of the second set of key-value pairs.
[0061] In block 212, the similarity score controller 180 determines a similarity score for each leaf node of the set of leaf nodes based on the corresponding leaf node eigenpoint and based on the object eigenpoint. The similarity score may be a numerical value along a predetermined scale. The similarity score may be determined based on or include a distance between the object eigenpoint and the leaf node eigenpoint in the multidimensional space. Additionally or alternatively, the similarity score may be determined based on or include a cosine similarity score between the object eigenpoint and the leaf node eigenpoint in the multidimensional space.
[0062] At block 214, the similarity score controller 180 identifies an incomplete subset of the set of leaf nodes in the decision tree based on the similarity scores. If the decision tree includes multiple trajectories that intersect or end at a single leaf node, identifying the incomplete subset can include identifying an incomplete subset of the trajectories in the decision tree.
[0063] The leaf nodes of an incomplete subset may include one or more leaf nodes.
[0064] Identifying the incomplete subset can include identifying each leaf node (or trajectory) associated with a similarity score that exceeds a predetermined or user-selected absolute or relative threshold. For example, the incomplete subset can include leaf nodes associated with a similarity score that exceeds 90% or 0.9. As another example, the incomplete subset can include leaf nodes associated with the top four similarity scores across leaf nodes.
[0065] In block 216, the decision tree monitor 110 obtains status or protocol information associated with each leaf node of the subset of the set of leaf nodes. For example, the status information may identify a particular disease, disease stage, disease subtype, disease progression, responsiveness to a given treatment class, etc. As an additional or alternative example, the protocol information may identify a recommended treatment (e.g., therapy). The treatment identification may identify a particular active ingredient, composition, route of administration, dosage and / or schedule. The information may be retrieved from (for example) a source of the decision tree, a source of files or information used to generate the decision tree, an external source, or an internal source. The information may be retrieved (for example) using a search function.
[0066] At block 218, the interface controller 165 generates output related to the subject, which includes state or protocol information or processed versions thereof. The output may further include information regarding the corresponding trajectory or trajectories leading to the leaf nodes in the subset. For example, for each decision node in a given trajectory, the output may identify the query and the corresponding value that was evaluated (or whether any corresponding value was available for evaluation).
[0067] The output can then be transmitted to a user device (eg, via a web page or electronic communication). IV. Working Examples IV.A. Example 1 - An Exemplary Challenge for Evaluating Decision Trees Using Structured Target Data
[0068] 3A and 3B show selected portions of the decision tree, and FIG. 4 shows the corresponding portion of the unstructured set of subjects. In FIG. 3A, the current traversal of the decision tree shows that a particular genetic mutation (ALK rearrangement) has been detected. The first decision to be made in the traversal of the depicted portion of the decision tree is whether this mutation was discovered before the first-line systemic treatment was administered or during such administration. If the rearrangement was discovered before the treatment, the traversal of the tree proceeds to a decision node to identify which first-line treatment was administered. On the other hand, if the rearrangement was discovered during the treatment, the traversal of the tree proceeds to a decision node to identify which treatment was administered after the planned systemic treatment was completed. The next decision node involves a query as to whether progression has occurred or not.
[0069] If an ALK rearrangement was found prior to first-line systemic therapy, the first-line therapy used was one of the following: alectinib, rigatinib, loratinib, or cerutinib. Once progression occurs, the traversal of the decision tree proceeds to the portion of the decision tree shown in Figure 3B. The next query to be evaluated is whether the subject is asymptomatic or symptomatic. If the latter, it should be determined whether the condition is a brain condition or systemic. If systemic, the next query is to characterize the condition as limited metastasis or multiple lesions.
[0070] While at least some of the responses to the decision node queries depicted in Figures 3A and 3B can be determined using the unstructured set of subjects depicted in Figure 4, it is difficult to determine these responses using automated techniques. For example, the structured dataset is longitudinal. Longitudinal data can also be diverse in terms of variables and size. Identifying protocols to reliably transform such diverse data in a meaningful way can be particularly challenging. Furthermore, identifying temporal relationships of events requires drawing from different fields to align various events with date and / or temporal order information. Furthermore, some of the data requested by the decision tree (e.g., whether a symptom is a systemic symptom or a brain symptom) is not available and must be inferred. Furthermore, the structured dataset is missing data (see, e.g., biomarker test date). Furthermore, as discussed above, the illustrated example of subject data is unstructured. Although protocols can be implemented to transform the data into structured data, attempting to determine which variables are important to track is difficult (given the complexity of the decision tree and the frequent changes to the decision tree) and does not address issues regarding missing data and challenges regarding determining the relative timing of various types of events.
[0071] Given that the target data and the decision tree may refer to different keys or labels, evaluating queries that depend on the structure of the decision tree may depend on the ability to map the information of the target data to variables identified in the decision tree. Figure 5A shows two examples of how target data can be mapped to decision tree variables based on the difference between how concepts are expressed. For example, "adenocarcinoma, large cell, NOS" is synonymous with "non-squamous". As another example, the illustrated example shows how structured data can include data that are differently expressed (e.g., "BRAF") or at different levels of precision (e.g., "BRAFV600E").
[0072] FIG. 5B shows two examples of how decision tree queries can relate to temporal sequences that require derivation using subject data or may not be accessible using the subject data. For example, the left table represents a decision tree query regarding the time that biomarker testing (to detect estimated glomerular filtration rate) was performed relative to the time that primary treatment was received. In this case, the date of biomarker testing was available in the subject data, as was the date that primary treatment was initiated. Although the subject data may lack an explicit identification of the time difference between events or the order in which events occurred, this information may be important for evaluating decision tree queries. For example, as shown in the right table illustrated, a temporal relationship query in a decision tree can be based on whether "progression" occurred on primary or subsequent treatment. As another example, a query can be based on whether "progression" occurred on a particular compound, "osimertinib". However, the subject data may present a date, such as the progression date, "2018-06-30". Logical and mathematical transformations can then be performed to standardize the subject data against the guideline queries. As yet another example, death in the subject data can also be marked as a progressive event and the date of death treated as the progressive date, resulting in a derived temporal relationship.
[0073] FIG. 5C shows an example where the decision tree query concerns a quantitative metric. However, the test results contained in the subject data include measurements and ranges that are capped at values different than the thresholds specified in the decision tree. Furthermore, the subject data includes categorical test results (e.g., failed / inconclusive tests) and results of a different granularity than those evaluated by the nodes of the decision tree. Thus, logic must be implemented to indicate how to handle situations where the thresholds specified in the decision tree are in the ranges identified in the subject data to evaluate the query.
[0074] Each of Figures 6A-6B illustrates a scenario of mapping subject data to values related to a query in a decision tree. Figure 6A illustrates a node in the decision tree that references (links) two principle pages, NSCLC-K1 of 5 and NSCLC-K2 of 5. The two principle pages contain content that has been curated and fed into a leaf node specific dataset. The two principle pages (not shown) contain information of the drug protocol used. If the hyperlinked phrase itself were used, the information in the linked page would be lost. Therefore, instead of treating NSCLC-K1 of 5 and NSCLC-K2 of 5 literally, text information from the pages of NSCLC-K1 of 5 and NSCLC-K2 of 5 can be extracted and added to the current version of the decision tree. In this way, the tokenization controller can tokenize the information from these two pages for further steps, thus enabling better matching.
[0075] FIG. 6B shows the information corresponding to the structured dataset and the mapped information for the converted navigation to the decision tree. Here, the query identified in the decision tree is whether the subject has multiple lesions. This information is not specifically present in the subject data, but given that the subject data indicates that there is a malignant neoplasm in each of the three regions (brain, spinal cord, and bone), it can be inferred that the subject has multiple lesions. Another query identified in the decision tree is whether the metastasis is "localized." Again, the fact that the subject data identifies neoplasms in different regions suggests that metastasis has occurred. However, "localized" is not defined in the guidelines. Thus, the mapping may define what qualifies as "localized" (e.g., by identifying a lower threshold of some body region in which the neoplasm is detected).
[0076] As yet another exemplary challenge, as shown in FIG. 6C, keys in a pathway leading to the identification of a specific treatment for adenocarcinoma may be very similar in relative frequency to keys in a pathway leading to the identification of a specific treatment for squamous cell carcinoma (e.g., a single key differs, i.e., "adenocarcinoma" vs. "squamous"). Thus, subjects diagnosed with these histological subtypes may be assigned or classified into the wrong pathway if tokenization and vectorization are applied indiscriminately to the two pathways, as the algorithm may confuse the two pathways. To address this possibility, the weight (relative frequency) of the adenocarcinoma pathway was increased in the adenocarcinoma key in the adenocarcinoma pathway to facilitate discrimination of the two pathways. IV.B. Example 2 -
[0077] We accessed a structured dataset for each of 36,469 subjects with aggressive non-small cell lung cancer. The subject data included biomarker test data indicating whether any of the six mutations were available in 55% of subjects (n=20,093) and unavailable in 45% of subjects (n=16,365).
[0078] For each subject, the subject data was converted into structured subject data including a first set of key-value pairs. The structured subject data was then converted into a second set of key-value pairs, where the keys of the second set of key-value pairs were different from the keys of the first set of key-value pairs. The keys of the second set of key-value pairs were selected by identifying decision tree terms (terms used to define queries at decision nodes) that were used inconsistently across multiple decision trees, whereby data indicating the relative frequency of the terms (term frequency scaled by inverse trajectory frequency) informs the selection of the keys. Specifically, n-gram tokenization was used to identify the keys or tokens (e.g., one-word, two-word, and three-word tokens). The representation of the structured data can then be converted into a numerical vector by projecting the tokens into a multidimensional space.
[0079] Figure 7 shows a preliminary evaluation of the algorithm's performance in matching a given subject to the correct decision trajectory. Two metrics are chosen to represent performance. The first accuracy metric is based on the number of nodes in the guideline path that are correctly matched. For example, if the subject's true decision tree trajectory consists of four nodes A→B→C→D, and the prediction is A→F→D, then the accuracy metric is Acc=2 / 3=0.666. This reflects that two predicted nodes were in the true path and the length of the prediction was used as 3 in the denominator.
[0080] The second accuracy metric is based on the Levenshtein distance, which is the minimum number of single edits (insertions, deletions or substitutions) required to change one sequence representing the true trajectory applied to the subject into the projected trajectory when applied to the subject. For example, if the subject's true decision tree trajectory consists of four nodes A → B → C → D, and the prediction consists of A → F → D, then the accuracy metric is Acc = 1 - 3 / 4 = 0.25. This means that at least three operations are required to convert the projected trajectory into the true trajectory: the deletion of F, the addition of B, and the addition of C. The 4 denotes the length of the ground truth, which is the longer between the ground truth and the projection used. The fraction (3 / 4) is then subtracted from 1 to generate the accuracy score.
[0081] To derive the precision scores in Figure 7, trajectories were manually identified for 35 subjects (5 subjects in 7 categories). The categories were chosen to reflect the distribution of subjects for the biomarker test results. Both types of precision scores were then calculated and averaged across the 5 subjects in that category to generate the numbers shown in the chart.
[0082] The scores themselves are not intended to be compared across existing algorithms, as the test cases here may lack generalizability. However, the scores may be useful for internal quality control purposes, so that the performance of the algorithm can be monitored during development. Furthermore, the scores can be evaluated to infer which categories of subjects the algorithm performs well or poorly for.
[0083] The first header row of the table shown in FIG. 7 identifies the selection keys of the second set of key-value pairs. Each of these keys can be associated with a "yes" or "no" label to indicate whether the biomarker was detected. The numbers in the second row identify, for each biomarker, the average count precision score of the predicted trajectories across five subjects in that category. The precision scores were calculated by assuming that the manually constructed trajectories are the true trajectories. The numbers in the third row identify the average Levenshtein precision score across five subjects in that category (also based on a comparison with the corresponding manually identified trajectories). V. Further Considerations
[0084] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product including instructions configured to cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein.
[0085] The terms and expressions employed are used as terms of description and not of limitation, and the use of such terms and expressions is not intended to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the claimed invention. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it should be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.
[0086] This specification provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the description of the preferred exemplary embodiments will provide those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.
[0087] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.
Claims
1. accessing a structured data set including a first set of key-value pairs, each of the first set of key-value pairs characterizing an assessment outcome or protocol characteristic of a subject; Transforming the first set of key-value pairs into a second set of key-value pairs, wherein at least some keys in the second set of key-value pairs are different from each key in the first set of key-value pairs; projecting the second set of key-value pairs to identify object unique points within a multi-dimensional space; accessing one or more decision trees including a plurality of decision nodes and a plurality of leaf nodes, each of the plurality of leaf nodes being connected to a root node via a leaf node-specific trajectory, each of the plurality of decision nodes corresponding to a criterion based on at least one value in the second set of key-value pairs; for each leaf node in the one or more decision trees, determining a leaf-node-specific point in the multi-dimensional space based on the leaf-node-specific trajectory; for each leaf node in the one or more decision trees, determining a similarity score based on the leaf node characteristic points and the object characteristic points; identifying an incomplete subset of the plurality of leaf nodes based on the similarity scores; obtaining state or protocol information associated with each leaf node in the incomplete subset; and generating an output related to the object including the status or protocol information; 4. A computer-implemented method comprising:
2. determining, for each leaf node in the one or more decision trees, the leaf node distinctive point; converting the leaf-node specific trajectories into a first leaf-node specific data set using text extraction; converting the first leaf-node-specific data set into a leaf-node-specific set of key-value pairs; and projecting a second leaf node unique set of key-value pairs to identify the leaf node unique point; The computer-implemented method of claim 1 , comprising:
3. The computer-implemented method of claim 1 , wherein determining the similarity score comprises applying a cosine similarity function.
4. determining, for each term of a term set, an inverse trajectory frequency indicating the frequency with which the term occurs across leaf-node-specific trajectories associated with leaf nodes in the one or more decision trees; determining, for each term in the term set, a term frequency indicating a frequency with which the term occurs in each of the leaf node-specific trajectories; The computer-implemented method of claim 1 , wherein values of the second set of key-value pairs are determined based on the determined reverse trajectory frequencies and the term frequencies.
5. The computer-implemented method of claim 1 , wherein the output further comprises, for each node in the incomplete subset, a query represented by a decision node in the leaf-node-specific trajectory.
6. The computer-implemented method of claim 1 , wherein the output comprises protocol information identifying potential treatments for the subject.
7. The structured data set includes, for each of the first set of key-value pairs, an initial timestamp, and converting the first set of key-value pairs to a second set of key-value pairs includes: classifying a particular key-value pair of the first set of key-value pairs as an indexing event; generating, for each key-value pair of the first set of key-value pairs, a modified timestamp using the initial timestamp associated with the key-value pair and the initial timestamp associated with the particular key-value pair; detecting that a decision node in the one or more decision trees includes a query to determine whether a particular event occurs within a particular time period relative to the occurrence of another particular event, the other particular event corresponding to the indexing event; executing a query to determine whether the first set of key-value pairs includes a first particular key-value pair that represents the particular event and that is associated with a modified timestamp within a particular time range; and determining a second particular key-value pair based on results of the query, the second set of key-value pairs including a second key-value pair; The computer-implemented method of claim 1 , comprising:
8. A computer program product tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform a set of operations, the set of operations comprising: accessing a structured data set including a first set of key-value pairs, each of the first set of key-value pairs characterizing an assessment outcome or protocol characteristic of a subject; Transforming the first set of key-value pairs into a second set of key-value pairs, wherein at least some keys in the second set of key-value pairs are different from each key in the first set of key-value pairs; projecting the second set of key-value pairs to identify object unique points within a multi-dimensional space; accessing one or more decision trees including a plurality of decision nodes and a plurality of leaf nodes, each of the plurality of leaf nodes being connected to a root node via a leaf node-specific trajectory, each of the plurality of decision nodes corresponding to a criterion based on at least one value in the second set of key-value pairs; for each leaf node in the one or more decision trees, determining the leaf node's unique point in the multidimensional space based on the leaf node's unique trajectory; for each leaf node in the one or more decision trees, determining a similarity score based on the leaf node characteristic points and the object characteristic points; identifying an incomplete subset of the leaf nodes based on the similarity scores; obtaining state or protocol information associated with each leaf node in the incomplete subset; and generating an output related to the object including the status or protocol information; 2. A computer program product comprising:
9. determining, for each leaf node in the one or more decision trees, the leaf node distinctive point; converting the leaf-node specific trajectories into a first leaf-node specific data set using text extraction; converting the first leaf-node-specific data set into a leaf-node-specific set of key-value pairs; and projecting a second leaf node unique set of key-value pairs to identify the leaf node unique point; 9. The computer program product of claim 8, comprising:
10. The computer program product of claim 8 , wherein determining the similarity score comprises applying a cosine similarity function.
11. The set of operations is determining, for each term of a term set, an inverse trajectory frequency indicating the frequency with which the term occurs across leaf-node-specific trajectories associated with leaf nodes in the one or more decision trees; determining, for each term in the term set, a term frequency indicating a frequency with which the term occurs in each of the leaf node-specific trajectories; 9. The computer program product of claim 8, wherein values of the second set of key-value pairs are determined based on the determined reverse trajectory frequencies and the term frequencies.
12. 10. The computer program product of claim 8, wherein the output further comprises, for each node in the incomplete subset, a query represented by a decision node in the leaf-node-specific trajectory.
13. 10. The computer program product of claim 8, wherein the output comprises protocol information identifying potential treatments for the subject.
14. The structured data set includes, for each of the first set of key-value pairs, an initial timestamp, and converting the first set of key-value pairs to a second set of key-value pairs includes: classifying a particular key-value pair of the first set of key-value pairs as an indexing event; generating, for each key-value pair of the first set of key-value pairs, a modified timestamp using the initial timestamp associated with the key-value pair and the initial timestamp associated with the particular key-value pair; detecting that a decision node in the one or more decision trees includes a query to determine whether a particular event occurs within a particular time period relative to the occurrence of another particular event, the other particular event corresponding to the indexing event; executing a query to determine whether the first set of key-value pairs includes a first particular key-value pair that represents the particular event and that is associated with a modified timestamp within the particular time range; and determining a second particular key-value pair based on results of the query, the second set of key-value pairs including a second key-value pair; 9. The computer program product of claim 8, comprising:
15. one or more data processors; 1. A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to: accessing a structured data set including a first set of key-value pairs, each of the first set of key-value pairs characterizing an assessment outcome or protocol characteristic of a subject; Transforming the first set of key-value pairs into a second set of key-value pairs, wherein at least some keys in the second set of key-value pairs are different from each key in the first set of key-value pairs; projecting the second set of key-value pairs to identify object unique points within a multi-dimensional space; accessing one or more decision trees including a plurality of decision nodes and a plurality of leaf nodes, each of the plurality of leaf nodes being connected to a root node via a leaf node-specific trajectory, each of the plurality of decision nodes corresponding to a criterion based on at least one value in the second set of key-value pairs; for each leaf node in the one or more decision trees, determining the leaf node's unique point in the multidimensional space based on the leaf node's unique trajectory; for each leaf node in the one or more decision trees, determining a similarity score based on the leaf node characteristic points and the object characteristic points; identifying an incomplete subset of the leaf nodes based on the similarity scores; obtaining state or protocol information associated with each leaf node in the incomplete subset; and generating an output related to the object including the status or protocol information; A non-transitory computer-readable storage medium for causing a set of operations to be performed, the set of operations including: Including, the system.
16. determining, for each leaf node in the one or more decision trees, the leaf node distinctive point; converting the leaf-node specific trajectories into a first leaf-node specific data set using text extraction; converting the first leaf-node-specific data set into a leaf-node-specific set of key-value pairs; and projecting a second leaf node unique set of key-value pairs to identify the leaf node unique point; The system of claim 15 , comprising:
17. The system of claim 15 , wherein determining the similarity score comprises applying a cosine similarity function.
18. The set of operations is determining, for each term of a term set, an inverse trajectory frequency indicating the frequency with which the term occurs across leaf-node-specific trajectories associated with leaf nodes in the one or more decision trees; determining, for each term in the term set, a term frequency indicating a frequency with which the term occurs in each of the leaf node-specific trajectories; 16. The system of claim 15, wherein values of the second set of key-value pairs are determined based on the determined reverse trajectory frequencies and the term frequencies.
19. 16. The system of claim 15, wherein the output further comprises, for each node in the incomplete subset, a query represented by a decision node in the leaf-node specific trajectory.
20. The system of claim 15 , wherein the output includes protocol information identifying potential treatments for the subject.
21. The structured data set includes, for each of the first set of key-value pairs, an initial timestamp, and transforming the first set of key-value pairs into a second set of key-value pairs; classifying a particular key-value pair of the first set of key-value pairs as an indexing event; generating, for each key-value pair of the first set of key-value pairs, a modified timestamp using the initial timestamp associated with the key-value pair and the initial timestamp associated with the particular key-value pair; detecting that a decision node in the one or more decision trees includes a query to determine whether a particular event occurs within a particular time period relative to the occurrence of another particular event, the other particular event corresponding to the indexing event; executing a query to determine whether the first set of key-value pairs includes a first particular key-value pair that represents the particular event and that is associated with a modified timestamp within the particular time range; and determining a second particular key-value pair based on results of the query, the second set of key-value pairs including a second key-value pair; The system of claim 15 , comprising:
22. A computer program product tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform a set of operations, the set of operations comprising: accessing a structured data set including a first set of key-value pairs, each of the first set of key-value pairs characterizing an assessment outcome or protocol characteristic of a subject; Transforming the first set of key-value pairs into a second set of key-value pairs, wherein at least some keys in the second set of key-value pairs are different from each key in the first set of key-value pairs; projecting the second set of key-value pairs to identify object unique points within a multi-dimensional space; accessing one or more decision trees including a plurality of decision nodes and a plurality of leaf nodes, each of the plurality of leaf nodes being connected to a root node via a leaf node-specific trajectory, each of the plurality of decision nodes corresponding to a criterion based on at least one value in the second set of key-value pairs; for each leaf node in the one or more decision trees, determining the leaf node's unique point in the multidimensional space based on the leaf node's unique trajectory; for each leaf node in the one or more decision trees, determining a similarity score based on the leaf node characteristic points and the object characteristic points; identifying the plurality of leaf nodes of an incomplete subset based on the similarity scores; obtaining state or protocol information associated with each leaf node in the incomplete subset; and generating an output related to the object including the status or protocol information; 2. A computer program product comprising:
Citation Information
Patent Citations
Homogeneous risk unit feature set generation method and device, equipment and medium
CN111126476A
Cloud-aided decision tree model diagnosis system and method for privacy protection of medical data
CN113517065A
Medical care information search device and program
JP2013054459A
Information recommendation device, recommendation information determination method, recommendation information determination program, and information recommendation program
JP2014191501A
Medical information retrieval system and medical information retrieval program
JP2015172838A
