Predictive models with decomposable hierarchical layers configured to produce interpretable results

Abstract

A computer-implemented method for providing interpretable predictions from machine learning models receives a data structure representing a hierarchical structure of a set of features (X) used by one or more predictive models to generate a set of predictions (Y). An interpretability model is constructed corresponding to the predictive models by assigning an interpretability to each prediction Yi of the predictive models based on the hierarchical structure. Assigning the interpretability includes decomposing X into multiple partitions Xj using the hierarchical structure, where X is represented by the following formula: [Equation 1] JPEG2023534956000013.jpg9170N is the number of partitions. Further, each partition is decomposed into multiple subpartitions from the multiple partitions using the hierarchical structure until an atomic subpartition is obtained. A score for each partition as a function of the predicted scores of the subpartitions of each partition is calculated, where the predicted scores represent interactions between the multiple subpartitions. Further, an interpretation of the prediction is output.

Description

[Background technology]

[0001] The present disclosure relates generally to programmable computers, and more specifically to a programmable computer system configured to generate and execute a set of predictive models that use a decomposable hierarchical structure of features grouped to different granularities to generate results that are more easily interpreted.

[0002] Many fields, such as the medical field, are attempting to incorporate the use of predictive machine learning models to perform tasks that require the analysis of data and further require the use of the analysis results as a basis for future actions. In general, predictive machine learning models are generated and executed by neural networks or other machine learning algorithms, which can be implemented as programmable computers configured to execute a set of machine learning algorithms. Neural networks incorporate knowledge from various fields, such as neurophysiology, cognitive science / psychology, physics (statistical mechanics), control theory, computer science, artificial intelligence, statistics / mathematics, pattern recognition, computer vision, parallel processing, and hardware (e.g., digital / analog / VLSI / optics).

[0003] The basic function of neural networks and other machine learning algorithms is to recognize patterns by interpreting unstructured data through a kind of mechanical perception. The unstructured real-world data in its native form (e.g., images, audio, text, or time series data) is converted into a numerical format (e.g., vectors with magnitude and direction) that can be understood and manipulated by a computer. The machine learning algorithm performs multiple iterations of learning-based analysis on the real-world data vectors until a pattern (or relationship) contained in the real-world data vector is discovered and learned. The learned pattern / relationship serves as a predictive model that can be used to perform various tasks, such as those mentioned above, including prediction of real-world data. Prediction tasks often rely on the use of labeled datasets to train neural networks (i.e., models) or other machine learning models to recognize correlations between labels and data. This is known as supervised learning.

[0004] While simple predictive models can be trivial using common linear models or decision tree models, the results of machine learning models generated for more complex tasks can be difficult to understand due to the more complex model structure. Machine learning models developed to perform complex tasks (e.g., sentiment analysis of online posts) are often "black box" models in that the relationship between the task assigned to the model and the results generated by the model is not clear. However, as tasks become more complex, the relationship between the model's results and the task the model was designed to perform can be unclear. Summary of the Invention [Means for solving the problem]

[0005] A computer-implemented method for providing interpretable predictions from machine learning models includes receiving a data structure representing a hierarchical structure of a set of features (X) used by one or more predictive models to generate a set of predictions (Y). The method includes constructing an interpretability model corresponding to the predictive models by assigning an interpretability to each prediction Yi of the predictive models based on the hierarchical structure. Assigning the interpretability includes decomposing the set of features (X) of the input data into a number of partitions Xj using the hierarchical structure, where X is represented by the following formula:

number

[0006] Other embodiments of the present disclosure implement features of the above-described methods in computer systems and computer program products.

[0007] Additional technical features and advantages are realized through the techniques of the present disclosure. Embodiments and aspects of the present disclosure are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, reference should be made to the detailed description and accompanying drawings.

[0008] The particular subject matter of the exclusive rights set forth herein is particularly pointed out and distinctly claimed in the claims appended hereto. The foregoing and other features and advantages of embodiments of the present disclosure will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates a machine learning system that can be utilized to implement aspects of the present invention. [Diagram 2] FIG. 2 illustrates a learning phase that can be implemented by the machine learning system shown in FIG. [Diagram 3] FIG. 3 illustrates a block diagram of an arrangement of machine learning models and components constructed and arranged in accordance with one or more embodiments of the invention. [Figure 4] FIG. 4 illustrates an illustration of an exemplary hierarchical structure in accordance with one or more embodiments of the present invention. [Diagram 5] FIG. 5 illustrates an illustration of an exemplary hierarchical structure example in accordance with one or more embodiments of the present invention. [Figure 6] FIG. 6 illustrates an example illustration of an attention model used to train predictions at a given granularity in accordance with one or more embodiments of the present invention. [Figure 7] FIG. 7 illustrates an illustrative example of a training model using a hierarchical structure in accordance with one or more embodiments of the invention. [Figure 8] FIG. 8 illustrates an illustrative example of quantifying the interaction of difference-based factors at the course level with the immediate next level predictions in accordance with one or more embodiments of the present invention. [Figure 9] FIG. 9 illustrates an exemplary illustrative example of an exemplary system for generating and executing predictive models that use a decomposable hierarchical structure of utilized features to generate results that are more easily interpreted in accordance with one or more embodiments of the present invention. [Figure 10]FIG. 10 illustrates an exemplary illustration of pseudocode for an algorithm for adding interpretability in a decomposable hierarchical structure in accordance with one or more embodiments of the present invention. [Figure 11] FIG. 11 illustrates a flow diagram of a method for providing interpretability to predictive models using a decomposable hierarchical structure in accordance with one or more embodiments of the invention. [Figure 12] FIG. 12 illustrates a block diagram of a processing system for providing interpretability to predictive models using a decomposable hierarchical structure in accordance with one or more embodiments of the invention.

[0010] The diagrams depicted herein are exemplary. There may be many variations in the diagrams, or in the operations depicted therein, without departing from the spirit of the disclosure. For example, the actions may be performed in a different order, or the actions may be added, deleted, or modified. Also, the word "connected" and variations thereof describe having a communication path between two elements, and do not imply a direct connection between the elements without an intervening element / connection between them. All of these variations are considered to be part of this specification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present disclosure is generally directed to a system and method for predictive models with decomposable hierarchical layers configured to generate interpretable results. The systems and methods described herein are directed to providing interpretable results that can be easily understood by a user, such as a decision maker in a specialized field, along with results from a machine learning model. The system can generate interpretable results from an input dataset generated by a machine learning model. The interpreted results provide context, such as a context that identifies factors that influenced a particular prediction, and present the context and the results of the machine learning model in a clear and easily understandable manner. The system utilizes a decomposable approach to add interpretability to each important factor of the results generated by the machine learning model. The interpreted results can be viewed at various granularities, allowing the user to zoom in to a more granular level where results are provided along with contextual reasoning and contextual justification.

[0012] An embodiment of the present disclosure is directed to providing interpretability to predictive models using a decomposable hierarchical structure and an attention neural network. A non-limiting exemplary computer-implemented method includes using a hierarchical structure created by one or more experts to generate multiple predictive models for each layer of the hierarchical structure. Each layer of the hierarchical structure includes a subset of features, where the root layer includes all of the features considered by the hierarchical structure. Each predictive model can use an attention neural network to predict an output using features from a corresponding layer. In one or more embodiments of the present invention, each layer is further divided into groups of features, and each partition is used to train an individual predictive model. By training such multiple predictive models from each layer, a final predicted output is generated with a score or value assigned to each partition.

[0013] One or more embodiments of the present invention include a computer-implemented method comprising partitioning input data according to the hierarchical structure. The method further comprises providing a predicted score or value for each partition. The method can further comprise decomposing each partition into subpartitions using the hierarchical structure. The method further comprises providing a new predicted score or value for each subpartition. The score can be calculated by using an attention neural network model. The method further comprises providing an interpretability of a prediction provided based on the input data using the contribution of the features (i.e., feature partitions) to the overall score or value of the predictive model. The interpretability is determined as an interaction of the scores or values ​​of the partition and its subpartitions.

[0014] The embodiments of the present invention include systems, computer program products, and other types of implementations. Thus, one or more embodiments of the present invention described herein are directed to providing interpretable results along with the results generated by the attention neural network model to provide context and clarity. By providing results that are more accessible and understandable than the results generated by the machine learning model, a user can take action for an identified situation based on a clear understanding of the results generated by the machine learning model.

[0015] In an aspect of the invention, the system can be configured to receive industry data (e.g., best practices, identification of attributes and features, relationships between features, etc.) from industry knowledge professionals (e.g., persons skilled in the art from an identified field). In an aspect of the invention, the system uses the industry data to build a hierarchical structure, which can be used by an interpretability model of the system to add interpretability to results generated by a machine learning model.

[0016] In some examples, an industry knowledge professional may identify and label a set of features or attributes or combinations thereof as a "partition." Partitions can also be developed using machine learning models by processing historical data. In the context of the present invention, the partition is configured as a set of closely related features. For example, in the medical field, a patient's partition may include features or attributes or combinations thereof, such as symptoms, medical history, age, sex, phone number, address, etc. In some examples, additional models may be used to represent the contribution from each partition, so that the entire set of features is represented by the hierarchical structure.

[0017] The systems and methods described herein can use a decomposable approach to add a layer of interpretability to the results generated by a machine learning model. In the context of the present invention, the decomposable approach allows a user to zoom in on a specific partition of data based on the results provided along with the reasoning and rationale provided by the hierarchical structure. The user develops future actions based on a clearer understanding of the results that can be interpreted according to the interpretation data provided using the hierarchical structure.

[0018] For the sake of brevity, conventional techniques associated with making and using aspects of the present invention may or may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs for implementing various technical features described herein are well known. Thus, for the sake of brevity, many conventional implementation details are only briefly mentioned herein or omitted entirely without providing details of well-known systems or processes or combinations thereof.

[0019] Many of the functional units described herein are labeled as modules. Embodiments of the present invention apply to a wide variety of module implementations. For example, a module can be implemented as a hardware circuit configured to include custom VLSI circuits or gate arrays; off-the-shelf semiconductors, e.g., logic chips, transistors, or other discrete components. A module can also be implemented in programmable hardware devices, e.g., field programmable gate arrays, programmable array logic, programmable logic devices. A module can also be implemented in software for execution by various types of processors. An identified module of executable code can comprise, for example, one or more physical or logical blocks of computer instructions, which can be organized, for example, as an object, procedure, or function. Nevertheless, the executable code of an identified module need not be physically located together and can include heterogeneous instructions stored in different locations that, when logically coupled together, comprise a module and achieve a predetermined purpose for the module.

[0020] In the following, with reference to the drawings, a detailed description of an exemplary system for providing interpretability to a predictive model using a decomposable hierarchical structure according to an embodiment of the present invention will be provided. An example of a machine learning technique that can be used to implement aspects of the present invention will be described with reference to FIG. 1 and FIG. 2. A machine learning model constructed and arranged according to an embodiment of the present invention will be described with reference to FIG. 3. FIG. 4 illustrates an example of an exemplary hierarchical structure. FIG. 6 illustrates an explanation showing an example of an attention model used to train predictions at a predetermined granularity. FIG. 7 illustrates an exemplary example of training a model using a hierarchical structure. FIG. 8 illustrates an exemplary example of quantifying the interaction of factors based on the difference between a course level of a prediction and the immediately following next prediction level. FIG. 9 illustrates an exemplary embodiment of a system for generating and executing a predictive model that generates results that are more easily interpreted using a decomposable hierarchical structure of utilized features. FIG. 10 illustrates an exemplary embodiment of pseudocode of an algorithm for adding interpretability using a decomposable hierarchical structure. FIG. 11 illustrates a flow diagram of a method for providing interpretability to a predictive model using a decomposable hierarchical structure. In addition, a detailed description of an exemplary computing system and network architecture capable of implementing one or more of the embodiments described herein will be provided with reference to FIG. 12.

[0021] FIG. 1 illustrates a block diagram showing a system 100 capable of implementing various aspects of the invention described herein. More specifically, the functionality of the system 100 is used in embodiments of the invention to generate various models (e.g., the predictive model 116 and the interpretability model 335 shown in FIG. 3). The system 100 includes a plurality of data sources 102 in communication with a predictor 110 over a network 104. In some aspects of the invention, the data sources 102 can bypass the network 104 and be fed directly to the predictor 110. The data sources 102 provide input data 120 that includes data / information inputs that will be evaluated by the predictor 110 according to embodiments of the invention. The data sources 102 also provide data / information inputs that can be used by the predictor 110 to train or update, or train and update, one or more models 116 generated by the predictor 110. The data sources 102 can be implemented as a wide variety of data sources, including, but not limited to, sensors configured to collect real-time data, data repositories (including training data repositories), and outputs from other predictors. The network 104 can be any type of communication network, such as, but not limited to, a local network, a wide area network, a private network, the Internet, or any of the above types of communication networks.

[0022] The predictor 110 can be a programmable computer, such as a computer system 1100 (shown in FIG. 12) that executes one or more algorithms. As shown in FIG. 1, the predictor 110 includes a suite of machine learning (ML) algorithms 112; and one or more models 116 that are relational (or predictive) algorithms generated (or learned) by the ML algorithms 112. The algorithms 112 or models 116 or combinations thereof of the predictor 110 are depicted separately for ease of illustration and explanation. In embodiments of the invention, the functions performed by the various algorithms 112 or models 116 or combinations thereof of the predictor 110 can be distributed differently than shown. For example, if the predictor 110 is configured to perform an overall task having subtasks, the suite of ML algorithms 112 can be segmented such that some of the ML algorithms 112 perform each subtask, as well as some of the ML algorithms 112 perform the overall task.

[0023] The ML algorithm 112 receives and evaluates input data (i.e., training data and data-under-analysis) from the data sources 102. The ML algorithm 112 includes the functionality necessary to interpret and utilize the format of the input data. For example, if the data sources 102 include image data, the ML algorithm 112 may include visual recognition software configured to interpret the image data. The ML algorithm 112 applies machine learning techniques to the received training data (e.g., data received from one or more data sources 102) to generate / train / update one or more models 116 that, over time, model the overall task and subtasks that the predictor 110 is designed to complete.

[0024] Referring now to FIG. 2, an example of a learning stage 200 performed by the ML algorithm 112 to generate the model 116 described above is illustrated. In the learning stage 200, the predictor 110 extracts features from the training data 205 and converts the features into a vector representation that can be recognized and analyzed by the ML algorithm 112. The training data 205 can be received as part of the input data 120 from one or more data sources 102. The feature vectors 210 are analyzed by the ML algorithm 112 to "predict" one or more outcomes using the training data and a target model (or the task of the model). The ML algorithm 112 facilitates revealing relationships between features in two training data 205 and between features in more than two training data 205. Examples of suitable implementations of the ML algorithm 112 include, but are not limited to, attention neural networks and general additive models. The learning or training performed by the ML algorithm 112 can be supervised in one or more embodiments of the present invention.

[0025] When the model 116 has been fully trained by the ML algorithm 112, a data source that generates “real world” data is accessed, and the “real world” data is applied to the model 116 to generate a usable version of the input data 120. In some embodiments of the invention, the input data 120 can be fed back to the predictor 110 and used by the ML algorithm 112 as additional training data 205 to update or refine, or update and refine, the model 116.

[0026] In the context of the present invention, the ML algorithm 112 and the model 116 can be configured to apply a confidence level (CL) to various ones of those results / determinations (including the input data 120) output by the predictor 110 in order to improve the accuracy of the specific results / determinations (130) output thereby. The ML algorithm 112 or the model 116 or a combination thereof, when making a determination as to whether the value of CL is below a predetermined threshold (TH) (i.e., CL < TH) or generating a result of being below, the prediction result 130 can be classified as having a “reliability” low enough to justify the conclusion that the prediction result 130 is not valid, and this conclusion can be used to determine when, how, or when and how the prediction result 130 is to be handled in downstream processing. In the case of CL > TH, the prediction result 130 can be considered valid, and this conclusion can be used to determine when, how, or when and how the prediction result 130 is to be handled in downstream processing. A number of different predetermined TH levels can be provided. The determination / result having CL > TH can be ranked from the highest CL > TH to the lowest CL > TH in order to prioritize when, how, or when and how the determination / result is to be handled in downstream processing.

[0027] In the context of the present invention, the predictor 110 can be configured to apply a confidence level (CL) to the input data 120. When the predictor 110 determines that the CL in the input data 120 is below a predetermined threshold (TH) (i.e., CL < TH), the input data 120 can be classified as being low enough to justify the classification of "unreliable" in the input data 120. When CL > TH, the input data 120 can be classified as being high enough to justify the determination that the input data 120 is valid. A number of different predetermined TH levels can be provided so that the input data 120 with CL > TH can be ranked from the highest CL > TH to the lowest CL > TH.

[0028] The functions executed by the predictor 110, more specifically by the ML algorithm 112, can be organized as a weighted directed graph, where the nodes are artificial neurons (e.g., modeling the neurons of the human brain), and the weighted directed edges connect the nodes. The directed graph of the predictor 110 can be organized such that one node forms an input layer node, one node forms a hidden layer node, and one node forms an output layer node. The input layer nodes are coupled to the hidden layer nodes, and the hidden layer nodes are connected to the output layer nodes. Each node is connected to all nodes in the adjacent layer by a plurality of connection paths, which can be depicted as directional arrows each having a connection strength. A plurality of input layers, a plurality of hidden layers, and a plurality of output layers can be provided. When a plurality of hidden layers are provided, the predictor 110 can perform unsupervised deep learning to perform one or more assigned tasks of the predictor 110.

[0029] Similar to the function of the human brain, each input layer node receives an input without adjusting the connection strength or performing a node sum. Each hidden layer node receives its input from all input layer nodes according to the connection strength associated with the relevant connection path. Similar multiplication of connection strengths and node sums are performed for the hidden layer nodes and the output layer nodes.

[0030] The weighted directed graph of the predictor 110 processes data records (e.g., output from the data source 102) one at a time, and the weighted directed graph "learns" by comparing any earlier prediction of a record with the known actual prediction of that record. Using a learning method known as "back-propagation," the error from the first prediction of the first record is fed back into the weighted directed graph of the predictor 110 and used to modify the weighted connections of the weighted directed graph a second time, and this feedback process continues for many iterations. During the training phase of the weighted directed graph of the predictor 110, the correct prediction for each record is known, and therefore the output nodes can be assigned a "correct" value. For example, a node value of "1" (or 0.9) is assigned to nodes corresponding to the correct class, and a node value of "0" (or 0.1) is assigned to other nodes. It is then possible to compare the computed values ​​of the output nodes of the weighted directed graph with this "correct" value and compute an error term for each node (i.e., the "delta" rule). These error terms are then used to adjust the weights in the hidden layer so that in the next iteration, the output values ​​will be closer to the "correct" values.

[0031] A technical challenge in existing systems implementing such predictive model-based data is that all features in the input data 120 are used to build a single predictive model. Because the predictive model is built using machine learning techniques, the interactions between multiple features in generating the predicted outcome 130 may not be easily discerned. Furthermore, in existing solutions, the predictive system 100 cannot be programmed to use the interactions between one or more features to generate the predicted outcome 130. An embodiment of the present invention addresses such technical challenge by using a hierarchical structure 113. The hierarchical structure 113 can be predefined by one or more experts or defined by curating data interpretations by one or more experts, where the experts include subject matter experts in the domain / area of ​​the input data 120 that the predictor 110 uses to generate the predicted outcome 130 based on the predictive model 116. In one or more embodiments of the present invention, the predictive model 116 is generated using the hierarchical structure 113. By using the hierarchical structure 113, the predictor 110 can further provide interpretability of the predictive model 116 that is used to generate the predicted outcome 130.

[0032] The hierarchical structure 113 may be input to the predictor 110 as part of the input data 120. Alternatively or additionally, the hierarchical structure 113 is input directly to the predictor 110, independent of the input data.

[0033] FIG. 3 is a block diagram of a machine learning model and an arrangement of components of the system 100 according to an embodiment of the present invention. FIG. 3 illustrates a different view of the system 100 shown in FIG. 1. The system 100 can be implemented in a variety of ways. As described in more detail below, one or more of the modules described herein, when executed by at least one processor of a computing device (e.g., computer system 1100 shown in FIG. 12), can enable the system 100 to provide interpretability to the predictive model using a decomposable hierarchical structure. For example, as described in more detail herein, the exemplary system 100 can receive historical data 310 from one or more data sources 102. The historical data 310 can include data collected from one or more systems in an industry, such as users; actions performed; and corresponding results; relationship data between features or attributes or between characteristics and attributes, etc. The historical data 310 can be collected from a specified time period, from different systems or entities, etc. In some examples, the historical data 310 can be used by the prediction module 315 to train one or more predictive models 116. The historical data 310 includes current data that can be used by the predictive models 116 executed by the prediction module 315 to generate one or more predictions.

[0034] In some examples, the prediction module 315 receives or accesses the historical data 310, or receives and accesses the historical data 310. The historical data 310 can be received by the prediction module 315 in predefined portions, or can be received as the historical data 310 is received by the system 100. In some examples, the prediction module 315 can train different types of prediction models 116, such as a multilayer perceptron (MLP) model, an attention model, a convolutional neural network (CNN), or any other machine learning technique. The prediction module 315 trains one or more prediction models 116 using training data 205 that includes expected predictions for particular input values. Although depicted separately from the input data 120, the training data 205 can be part of the input data 120. Each prediction model 116 operates on a subset of the features of the input data 120 to generate an individual prediction. The predictions can be combined to generate a final prediction result 130. Alternatively or additionally, in one or more embodiments of the invention, one of the prediction models 116 operates on all of the features to generate the final prediction result 130, while the other prediction models 116 are used to determine the interpretability of the prediction result 130. The separation of the features among the prediction models 116 is done based on a hierarchical structure 113.

[0035] In one or more embodiments of the invention, the prediction module 315 uses the expert hierarchy 113 of the system 100 to generate an interpretability model 335. The interpretability model 335 is trained to determine interaction scores between one or more predictive models 116. In turn, the interpretability model 335 is trained to determine interaction scores for a subset of features used by each predictive model 116.

[0036] Once the predictive model 116 and the interpretability model 335 are trained, the prediction module 315 is used to generate predicted outcomes 130 using “live” data, i.e., input data 120 where the outcome is unknown. For example, users from an industry may provide industry data as input to the system 100. The industry data may be stored in a data store, or may be provided directly, or may be stored in a data store and provided directly. The industry data may include industry best practices, characteristics and attributes of various aspects of the industry (e.g., user demographics, transactions, interactions, financial information, risk assessments, etc.), etc. The industry data 305 may also include relationship information between the information provided.

[0037] The system 100 receives industrial data or access to industrial data, and generates model inputs with different granularity for the predictive model 116 based on the hierarchical structure 113. In some examples, the hierarchical structure 113 is a data structure, such as a tree structure. Each level of the tree structure includes one or more partitions. Each partition is a set of features that belong to the same data source or have similar characteristics (e.g., industry characteristics, data attributes, etc.) or belong to the same data source and have similar characteristics. It is understood that the partitioning can be based on various factors that can vary in different embodiments, and not all such factors are described herein. Different levels of the hierarchical structure 113 provide different granularity of the same data. The features at the top of the tree structure of the hierarchical structure 113 can be a combination of all available features from different partitions. The levels of the hierarchical structure 113 are specified by a user, or based on the need for depth of inference to provide context for the target results generated by the machine learning model, such as the predictive model 116, or a combination thereof.

[0038] In one or more embodiments of the invention, the interpretability model 335 includes a set of attention prediction models that use the hierarchical structure 113 to generate interpreted data 340 based on the input data 120. The interpreted data 340 includes information associated with the predicted scores or outputs from the set of predictive models 116. The interpreted data 340 may further include the partition of features in the input data 120 and interpretable inferences associated with forming the partition of features based on the hierarchical structure 113, predicted scores or values ​​generated for one or more partitions (and / or subpartitions) of features or attributes or combinations thereof identified in the input data 120 based on the hierarchical structure, identification of factors that influenced the input data 120, etc. The interpreted data 340 may provide scores associated with features or partitions in the hierarchical structure, as well as contextual inferences and explanations to provide additional context to the partitioning of the input data 120. A user of the system can use the interpreted data 340 to create interpretable attentions (e.g., weights) and then identify appropriate controllable actions from the attentions (or weights) as described herein.

[0039] 4 illustrates an illustration of an example hierarchical structure 113 in accordance with one or more embodiments of the present disclosure. The hierarchical structure 113 is a data structure used to provide interpretability at all levels of the hierarchy using a decomposable approach applied to input data 120 that is used by the predictive models 116 to generate prediction results 130. In some examples, the hierarchical structure 113 is based on industry best practices, which can be provided by an industry user as part of the industry data provided to the system 100, or can be generated from historical data 310 using one or more ML algorithms 112, or a combination thereof can be performed.

[0040] At each layer of the hierarchical structure 113, there are several partitions or groups of features. Each feature is associated with a score for predicting an outcome. The overall score of the predicted outcome 130 can be a weighted average of the results from each partition (and subpartitions) of the feature.

[0041] In the example shown in FIG. 4, the root node 410 of the hierarchical structure 113 shows all the features, which are partitioned into three partitions at the lower levels of the tree data structure: a medical history partition 420, a patient profile partition 430, and a patient behavior profile partition 440. Each partition is the parent of a set of related features. For example, the medical history partition 420 is the parent of the following features: a diagnosis partition 422, a family history partition 424, and a medication management partition 426. The patient profile partition 430 includes a person partition 432 (which is the parent of an age attribute 433 and a gender attribute 434) and an account partition 436 (which is the parent of an insurance attribute 437 and an office visit attribute 438). The patient behavior profile partition 440 is the parent of a follow-up partition 442 and an interaction partition 444. Each of the different levels of the hierarchical structure 113, i.e., 450, 460, and 470, represents a different level of granularity and detail. It should be noted that a "level" is different from a "partition" and that a level can include one or more partitions. For example, one of the partitions in level 450, the patient profile partition 430, contains different aspects of the patient, such as the patient's demographic information and the patient's account information. At level 450, all characteristics related to the patient are represented by the patient profile partition 430, but as the information is divided into lower levels (e.g., levels 460 and 470), the information about the patient becomes more detailed. For example, level 470 stores the patient's age and gender, and indicates the specific value of the patient's age or gender, respectively. The number of levels in the hierarchical structure can be designed by the user during the generation of the hierarchical structure and can depend on the need for depth of inference to provide a context for the prediction results 130 output by the predictive model 116.

[0042] It should be noted that the hierarchical structure 113 shown in FIG. 4 is one possible example in an exemplary scenario of patient data, and various other representations of the hierarchical structure 113 can be generated in the same exemplary scenario. Furthermore, different hierarchical structures can be created in other exemplary scenarios, such as financial data, business operation data, image recognition data, automated driving data, or any other type of data to be processed by machine learning. In some examples, a user can add a new partition to the hierarchical structure 113, for example, by instantiating an existing partition using a different dataset.

[0043] FIG. 5 illustrates a visual representation of a hierarchical structure 113 according to one or more embodiments of the present invention. The hierarchical structure 113 is another view different from the view of the structure illustrated in FIG. 4. Here, a representation is also provided to show that all levels (L0-L4) provide different granularity for the same data, and that each level can be partitioned in a different manner. For example, in FIG. 5, L0 represents the top layer without any partitions. Further, multiple layers, such as L2, L3, and L4, have partitions. The bottom layer, L4, includes atomic features f_01, f_02, ..., f_13. Here, "atomic features" represent data that is not further decomposed / refined. It should be noted that the number of partitions, the number of subpartitions, and the number of layers can vary in other examples.

[0044] In this example, there are four levels in the hierarchical structure, namely L1, L2, L3, and L4. Each level has a corresponding number of partitions, which are feature groups. Each level has a respective attention model as a predictive model from among the predictive models 116. Each individual predictive model 116 utilizes the same input data, but grouped to a different granularity, to generate an output score. The output score of the attention model for each level represents the weighting of each partition from that level. For a given instance of input data 120, each predictive model generates a score for each level in the hierarchical structure 113. The overall score is the weighted average of the weighted scores of each group. Atomic features f_01 to f_13 are illustrated at the lowest level (L4 and below). The partitions at each of the levels L1 to L4 group these features in different combinations. The exemplary groupings shown in FIG. 5 can be represented as provided in Table 1 herein.

[0045] [Table 1]

[0046] The first level, L1, contains all of the features f01-f13 in a single group, while the second level, L2, splits the features into three groups, the third level, L3, splits the features into five groups, and the fourth level, L4, contains seven groups. As can be seen above, the groups at some levels split the features in a different way than the splits at other levels.

[0047] FIG. 6 illustrates an example of an attention model 500 used to train predictions at a given level of granularity, according to one or more embodiments of the present disclosure. In some examples, the interpretability model 335 can be the attention model 500 used to train predictions at a specified level of granularity of the hierarchical structure 113. In some examples, the interpretability model 335 can be an attention model 500 where the input variables 510 can be an expert network, e.g., expert networks 505A, 505B, 505C (collectively referred to as 505). The expert network 505 can be a neural network. In one or more embodiments of the present invention, the expert network 505 is the prediction model 116. The attention model 500 can use a smooth soft attention function using a gating network. The input variables 510 of the attention model 500 can include multiple variables for a given expert network. Each expert network 505 represents an input attribute or a group of input attributes. In some examples, the input variables 510 can be provided as one of a number of arrays. The expert network 505 can be input to one or more activation functions 515A, 515B, and 515C (collectively referred to as 515). The activation function can define an output given an input or a set of inputs, e.g., the expert network 505. Examples of activation functions 515 include, but are not limited to, a tanh function, a rectified linear unit (ReLU) function, and a softmax function (also known as softargmax or a regularized exponential function). The output of the activation function 515 can include many variables to be predicted by the attention model 500. The attention model 500 can use hyperparameter autotuning 520 on the output of the activation function 515.Hyperparameter auto-tuning 520 can be performed using an ensemble of techniques that can include grid search, random search, and exponential learning using a validation dataset. For training purposes, the interpretability model 335 can use best hyperparameter values ​​and a stochastic gradient approach as well as multiple training streams to handle randomness in initialization. In one or more embodiments of the present invention, the interpretable results are calculated based on at least: 1) the model's score or value for a single feature without considering the scores / influence of other grouped features, and 2) the contribution of each set of grouped features as a product of the score (value) and the corresponding weight. The interaction of the feature groups is calculated by a comparison of the output of the two models from two levels of feature grouping (lower levels are more partitioned and different layers are grouped together).

[0048] FIG. 7 illustrates an illustrative example of training a model 600 using a hierarchical structure in accordance with one or more embodiments of the present invention. In some examples, the model 600 is trained using an attention modeling approach and a partition and hierarchical structure 113 to decompose the features into partitions and provide interpretability to the partitions and features. Prediction using the trained model can use a multi-layer perceptron. By training the model and performing the decomposition on the features, the system can provide the user with interpretability of the features. Decomposing the features into partitions at different levels allows for identifying interactions between partitions and levels of features and how partitions or features or combinations thereof affect different levels of features (e.g., level 1, level 2, etc. in FIG. 5). In some examples, the interaction of the partitions of features is calculated by comparing the output of two models for two levels of partition grouping (e.g., the lower layer is further partitioned, etc.).

[0049] Level 1 of the model (L1 in FIG. 5) represents all features of the training model. The features are decomposed into two or more partitions of features in L2 (FIG. 5). In this case, two partitions are shown: E1 610 and E2 620. All features are represented by node 605 at level 1. The features are partitioned into E1 610 and E2 615 at level 2. Using the three partitions in this setup, individual predictive models 116 are trained: a first predictive model for partition 605 for all features, a second predictive model for partition E1 610, and a third predictive model for partition E2 615. Furthermore, the attention model can use an ensemble approach for the different decomposed partitions to obtain an overall score for the entire feature set, while ensuring that the score for each partition is generated to represent its influence or interaction value in the learning model. The attention model represents an interpretability model 335. The overall score can be calculated by applying weights to the different partitions (e.g., 0.4 for E1 610 and 0.6 for E2 615). In some instances, the weights are not constant and are changed or modified for different reasons (e.g., client needs, targeted objects, etc.). The training model can repeat the process for each level, up to L4 in the exemplary hierarchical structure 113 in FIG. 5, until there are no more factors to decompose (reference number 620). It should be noted that the weight values, number of partitions, and other details shown in the figures herein are exemplary and that in other embodiments, such values ​​can vary.

[0050] FIG. 8 illustrates an illustrative example 700 of quantifying the interaction of multiple factors based on the difference at a course level with the immediately next level prediction, according to one or more embodiments of the present invention. In some examples, the system 100 can train an interpretability model 335 using the hierarchical structure 113. The interpretability model 335 quantifies the interaction of two or more factors (x1 and x2), where "factors" can include a group of features for which an interaction is determined. For example, the interpretability model quantifies the interaction between partition E1 610 and partition E2 615 (see FIG. 7). The level 1 node 710 represents the entire set of input data 120 used by the prediction model 116. The input data 120 can be partitioned into different partitions, namely E1 715 and E2 720. As illustrated in table 702, level 1 represents the input data 120, and level 2 represents partitions E1 715 and E2 720. Each partition of data is processed to determine the results resulting from that partition and to determine the associated interpretability. In one or more embodiments of the invention, the interpretability module 335 used to determine and associate such interpretability with a first partition can be different from the interpretability model 335 used for a second partition. That is, a set of interpretability models 335 can be used to discover the interpretability and interactions of features in the input data 120. For example, the set includes a separate interpretability model 335 for each partition. Alternatively, each partition can be associated with interpretability information using multiple interpretability models 335 from the set.

[0051] In the example scenario of FIG. 8, the hierarchical structure has two layers, namely L1 and L2. The input data 120 is processed using model h0(x1,x2) and attention (or weight) 1.0. E1 715 of layer L2 is processed using model f1(x1) and attention (weight) w1, while E2 720 of L2 is processed using model g1(x2) and attention w2. Here, h0, f1, and g1 represent the prediction models 116 associated with those partitions. The interaction between the two partitions E1 715 and E2 720 can be shown in the following equation:

number

[0052] FIG. 9 illustrates an illustrative example of a system for generating and executing predictive models that use a decomposable hierarchical structure of utilized features to generate results that are more easily interpreted, according to further embodiments of the present invention. The system 100 is considered to have the predictive models 116 trained to generate individual predictions based on the groups of features on which they are trained. Furthermore, an interpretability model 335 is also trained to determine an interaction score between each partition at each level in the hierarchical structure 113. The industrial data is used as input data 120 to the system 800. The industrial data 305 can be stored in a data store or can be provided directly to the system 100, or can be stored in a data store and provided directly to the system 100. The industrial data can include relationship information between the feature information provided, and is structured in a predetermined manner that the system 100 is trained to process and generate the prediction result 130. The hierarchical structure 113 is also provided to the system 100 to partition the input data 120 so that interpretability can also be provided along with the prediction result 130.

[0053] The prediction module 315 uses one or more predictive models 116 to process the input data 120 to generate predictions and other results according to training. Predictions from the prediction module 315 can be difficult to understand despite their accuracy and may not provide context about how one or more features or groups (i.e., partitions) of features, or combinations thereof, in the input data 120 are used by the predictive models 116 to generate the predicted results 130. Alternatively or additionally, it may be difficult to interpret what the predicted results 130 show.

[0054] In some exemplary embodiments, the system 100 can receive input data 120 and add interpretability. The system 100 can partition the input data 120 generated by the prediction module 315 into partitions of features according to the hierarchical structure 113. The input data 120 is partitioned separately for each of the predictive models 116. As described above, each partition at each level of the hierarchical structure 113 has its own corresponding predictive model 116. Further, each partition can be divided into subpartitions at the next (subsequent) level, up to the last level where the features are considered atomic. Thus, the input data 120 is further partitioned into mutually exclusive sets of features up to the last level in the hierarchical structure 113. All partitions, when taken together (e.g., union operation), provide a complete coverage of the features of the input data 120.

[0055] The interpretability model 335 at each level of the hierarchy can be an attention model that can provide a prediction score for each partition. Depending on whether the partitions are generated using classification or regression techniques, the interpretability model 335 can generate a score for each partition of grouped features. The prediction score of a partition (i.e., a group of features) can be interpreted as the quantified influence of that partition on the final prediction result 130 without considering other features in the input data. The interpretability model 335 can generate an overall score for the prediction result 130 based on a weighting approach, where each of the partitions (and subpartitions) is weighted, and all the weighted scores are used to calculate the overall score. The weights applied to the partitions can be based on different factors and are non-negative values ​​for each of the partitions. In some examples, the sum of the different weights used for different partitions is equal to 1. In some examples, the contribution of a given partition of features to the overall prediction result 130 is its weight multiplied by its score.

[0056] As illustrated in FIG. 9, the system 100 receives input data 120. All features from the input data 120 are represented at level L1 y0 815, which includes all features of the input data 120. The system 100 processes the input data 120 for each level in the hierarchical structure 113, for example, by a predictive model or an interpretability model 335, or a combination thereof. In one or more embodiments of the invention, each level in the input data 120 is processed iteratively according to each level in the hierarchical structure 113. The input data 120 is decomposed into mutually exclusive subpartitions at level y1 820, and as such processing occurs, a score is generated for each partition or subpartition. The system 100 processes the input data 120, decomposing the features into partitions and subpartitions at different levels (e.g., y0 815, y1 820, y2 830, etc.), and generates a score for each partition (and subpartition) until the feature cannot be decomposed. The scores and feature partitions are used to provide interpretability of the input data 120 at different levels of granularity based on the different decomposed partitions and sub-partitions generated from the input data 120.

[0057] 10 illustrates an illustrative example of pseudocode 900 for an algorithm for adding interpretability in a decomposable hierarchical structure in accordance with one or more embodiments of the present invention. The pseudocode 900 illustrates how the decomposition of features into partitions is performed and how interactions between multiple partitions are computed. The interactions provide interpretability to the features and how different partitions can affect other partitions and the overall score of the feature set.

[0058] The algorithm can be a tree walk for each hierarchy 113 with a breadth-first traversal from top to bottom. The first step of the algorithm can include using the hierarchy 113 as an expert model and partitioning the training data 205 according to the hierarchy 113. In the next step, the training data 205 is used to build a predictive model 116. Each partition can be assigned a Y, where Y is the TRUE target or expected prediction outcome for that partition. Consider a partition contains a feature set of the following formula:

number

[0059] The third step can be an iterative step to assign interpretability to a predictive model 116. For example, a predictive model 116 is selected for the top partition (containing all of one or more features) and has the following known outcomes:

number

number

number

number

[0060] The above process is repeated for each partition until the last level in the hierarchy 113 is reached.

[0061] In some exemplary embodiments, a top-to-bottom approach is used where the current model with interpretability is utilized where an additive (or logarithmic additive) modeling approach is used. Each factor (e.g., partition of features) contributes to a final score or value (e.g., prediction or probability). The contribution of each factor is a multiplication of the weight and the score associated with each factor. In some examples, the weight used for each factor can be a constant value. In some examples, the weights can be dynamically adjusted based on different needs of clients or samples.

[0062] The weakness of the top-to-bottom approach can be that additive models typically have a constrained model architecture, which can limit the accuracy of the model. If there are too many factors (i.e., partitions) involved, the contribution of each factor can be negatively influenced by the other factors. In some cases, it can be difficult to identify the aggregated influence (e.g., interactions) of multiple factors that belong to the same category.

[0063] In some exemplary embodiments, the weaknesses of additive models can be mitigated by including the interactions of multiple factors with a hierarchical structure 113. The additive models can be used with the granularity required by industry knowledge professionals.

[0064] In some examples, the system 100 determines the choice of input data 120 used to train the interpretability model 335. For example, in a first choice, the system 100 chooses to use the results of a previously trained model, which removes outliers from the statistical results, but moves away from the true (accurate) results. In a second choice, the system 100 chooses to use the results of the MLP training model, which provides the most accurate results from a statistical evaluation using all the features of the input data 120, reflects predictions based on all known knowledge, and removes some noise from the TRUE target. However, the results may contain some "noise" or outliers. A third choice may allow the interpretability model 335 to use the true target data, which ensures that all features of the partition have scores or predictions that are aligned to the TRUE target. However, the results may contain some "noise" or outliers.

[0065] 11 is a flow diagram of another exemplary computer-implemented method 1000 for providing interpretability to a predictive model having decomposable hierarchies configured to generate interpretable results. The steps shown in FIG. 11 can be performed by any suitable computer-executable code or computing system or combination thereof. In one example, each of the steps shown in FIG. 11 represents an algorithm whose structure includes or is represented by multiple sub-steps, or represents an algorithm whose structure includes and is represented by multiple sub-steps, examples of which will be provided in more detail below.

[0066] As illustrated in FIG. 11 , at step 1002, one or more of the systems described herein may receive the hierarchical structure 113. The systems described herein may perform step 1002 in any suitable manner. In some examples, the system 100 receives data, e.g., industrial data, from one or more data sources in association with the hierarchical structure 113 to be used to analyze the input data. Alternatively or additionally, a user provides information regarding one or more features to be included in one or more partitions in the hierarchical structure 113. The hierarchical structure 113 may be curated by one or more experts, thereby based on information to be derived from industrial data, which may contain information, e.g., industry best practices, configurations for identified use cases, and user information, relationships between attributes or features, or combinations thereof, etc. The hierarchical structure 113 may be received and stored as a data structure used to store a tree structure. The hierarchical structure 113 may identify how to divide the input data into partitions that include sets of features, and create associations between the partitions.

[0067] In some embodiments, the hierarchical structure 113 is a data structure constructed using a set of attributes. For example, based on the type of input data 120 to be analyzed by the system, the features in the hierarchical structure 113 can include a set of demographic features, financial features, or transactional features, or a combination thereof. Features are a set of closely related or associated parameters obtained from a dataset. In some examples, the hierarchical structure 113 can be, for example, a tree structure, where each level of the tree can include one or more partitions. Each partition can include a set of features that belong to the same data source or possess similar properties, or that belong to the same data source and possess similar properties. Different levels of the hierarchical structure 113 are represented as a set of partitions of data at different levels of granularity. For example, as illustrated in Figures 4 and 5, higher levels in the tree are broader classifications of data, while lower levels of the tree are finer granularity of data. Thus, the highest node illustrated in Figures 4 and 5 represents the union of all features available from the dataset.

[0068] 11 , at step 1004, the method 1000 includes receiving input data 120 to be analyzed using one or more predictive models 116. In some examples, at least one of the predictive models 116 implemented by the prediction module 315 can be an MLP model.

[0069] At step 1006, one or more systems described herein partition the input data into groups of features according to a hierarchical structure 113.

[0070] In some examples, the input data 120 is decomposed into mutually exclusive sets of features using the hierarchical structure 113. The partitioning of the input data is based on the feature groups defined at each given level of the hierarchical structure 113. The system 100 also iteratively decomposes the input data 120 into a specified number of levels of the hierarchical structure 113.

[0071] At step 1008, predicted scores and contextual rationales are determined for each partition. The systems described herein may perform step 1008 in any suitable manner. For example, the interpretability model 335 generates individual predicted scores and contextual rationales for each partition of features from the input data 120. The predicted scores and corresponding contextual rationales may augment the input data 120 to provide contextual information to enhance the interpretability of the prediction results 130. By applying the hierarchical structure 113 to the input data 120 while processing the input data 120 with the interpretability model 335, the input data 120 may be organized and expanded so that a user may identify a particular level within the result, and may also obtain contextual data that explains how the features in the input data 120 interact with each other, which factors influenced the prediction results 130, and provides additional data, e.g., predicted scores and explanations, for the identified partitions of features.

[0072] Each partition is decomposed into feature subpartitions at step 1010. The systems described herein may perform step 1010 in any suitable manner. For example, the interpretability model 335 decomposes the partitions into feature subpartitions based on feature groups defined at a given layer of the hierarchical structure 113.

[0073] At step 1012, a score is calculated for each subpartition and partition in the hierarchical structure 113. The systems described herein may perform step 1012 in any suitable manner. For example, the interpretability model 335 may generate individual predicted scores or values ​​and contextual rationale for each subpartition of features from the parent partition. The interpretability model 335 may return to step 1010 and iteratively decompose further into subpartitions of features based on the hierarchical structure 113 until no more features can be partitioned.

[0074] In some examples, the system 100 identifies a recommended action based on a predicted score or value for the selected partition or input data 120, or a combination thereof, and facilitates execution of the action based on user input. For example, if the predicted score or value indicates that the patient is eligible for a medical trial for a particular diagnosis, the system can present the patient with a suggested action, such as generating an electronic communication (e.g., email) notifying the patient that the patient is eligible for the trial, outlining the particular type of medical treatment available through the medical trial. If the user selects to proceed with the action, the system can generate and send the electronic communication to the patient.

[0075] The present invention provides a system and apparatus for providing interpretability to one or more predictive models using a decomposable hierarchical structure. The present invention can facilitate interaction between industrial knowledge workers and data scientists to generate models that have interpretability of the results of a predictive model or models that can identify the interpretability. The present invention also provides an expert model that is generated by an expert, and identifies and generates a hierarchical structure with appropriate allocation and partitioning of features. The present invention also provides an expert model that is generated by an expert, and enables an interactive process to add / remove / update important factors (features or group features) from the hierarchical structure. The present invention also provides a system and apparatus for providing interpretability to one or more predictive models using a decomposable hierarchical structure. The present invention also provides an expert model that is generated by an expert, and uses the features of this local level to generate interpretability of the results to generate an interpretable model ...

[0076] The disclosed embodiments of the present invention facilitate providing an interpretability model based on the contribution of features or feature partitions to the result scores and values. A decomposable hierarchical structure of features exists as feature partitions defined by a domain expert. A predictive model for each layer of the feature hierarchy is generated using an attention neural network using the same training data. A predicted score or value is provided for each partition of the feature partitions using the attention neural network model. Each partition of the feature partitions is decomposed into subpartitions using the hierarchical structure. A new predicted score or value is provided for each subpartition of the subpartitions using another attention neural network model. By applying the attention model using iterations on the decomposable hierarchical structure, the contribution of each partition of features at different granularities is identified. The interaction contribution of the feature partitions is identified through a comparison of the scores of a partition and its subpartitions.

[0077] Turning now to FIG. 12, a computer system 1100 according to an embodiment of the present invention is generally illustrated. The computer system 1100 can be an electronic, computer framework that includes or uses any number and combination of computing devices and networks utilizing various communication technologies as described herein, or that includes and uses. The computer system 1100 can be easily scalable, extensible, and modular, with the ability to be modified into different services or reconfigured in some functions independently of others. The computer system 1100 can be, for example, a server, a desktop computer, a laptop computer, a tablet computer, or a smartphone. In some examples, the computer system 1100 can be a cloud computing node. The computer system 1100 may be described in the general context of computer system executable instructions, such as program modules, being executed by the computer system. Generally, a program module may comprise routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system 1100 may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

[0078] As shown in FIG. 12, computer system 1100 includes one or more central processing units (CPUs) 1101a, 1101b, 1101c, etc. (collectively or generally referred to as processors 1101). Processor 1101 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Processor 1101, also referred to as a processing circuit, is connected to system memory 1103 and various other components via a system bus 1102. System memory 1103 may include read only memory (ROM) 1104 and random-access memory (RAM) 1105. ROM 1104 is connected to system bus 1102 and may include a basic input / output system (BIOS) that controls certain basic functions of computer system 1100. RAM is read-write memory connected to system bus 1102 for use by processor 1101. System memory 1103 provides temporary memory space for the execution of the instructions during operation and may comprise random access memory (RAM), read only memory (RAM), flash memory, or any other suitable memory system.

[0079] Computer system 1100 includes an input / output (I / O) adapter 1106 and a communications adapter 1107 connected to a system bus 1102. I / O adapter 1106 may be a small computer system interface (SCSI) adapter that communicates with a hard disk 1108 or any other similar component or combination thereof. I / O adapter 1106 and hard disk 1108 are collectively referred to herein as mass storage 1110.

[0080] Software 1111 for executing on computer system 1100 may be stored in mass storage 1110. Mass storage 1110 is an example of a tangible storage medium readable by processor 1101, where software 1111 is stored as instructions for processor 1101 to operate computer system 1100 as described herein below with respect to various figures. Examples of computer program products and the execution of such instructions are described in more detail herein. Communications adapter 1107 interconnects system bus 1102 with network 1112, which may be an external network, enabling computer system 1100 to communicate with other such systems. In one embodiment, a portion of system memory 1103 and mass storage 1110 collectively store any suitable operating system, which may be, for example, z / OS or AIX operating systems from IBM Corporation, for coordinating the functions of the various components shown in FIG. 12.

[0081] Additional input / output devices are shown connected to the system bus 1102 through a display adapter 1115 and an interface adapter 1116. In one embodiment, adapters 1106, 1107, 1115, and 1116 may be connected to one or more I / O buses that are connected to the system bus 1102 through an intermediate bus bridge (not shown). A display 1119 (e.g., a screen or display monitor) is connected to the system bus 1102 by a display adapter 1115, which may include a graphics controller and a video controller to improve the performance of graphics-intensive applications. A keyboard 1121, a mouse 1122, speakers 1123, etc. may be interconnected to the system bus 1102 through an interface adapter 1116, which may include, for example, a super I / O chip that integrates multiple device adapters into a single integrated circuit. Suitable I / O buses for connecting peripheral devices, such as hard disk controllers, network adapters, and graphics adapters, typically include a common protocol, such as Peripheral Component Interconnect (PCI). Thus, as configured in Figure 12, computer system 1100 includes processing capability in the form of a processor 1101, storage capabilities, such as system memory 1103 and mass storage 1110, input means, such as keyboard 1121 and mouse 1122, and output capabilities, such as speakers 1123 and display 1119.

[0082] In some embodiments, the communications adapter 1107 can transmit data using any suitable interface or protocol, such as an Internet Small Computer System Interface, among others. The network 1112 can be a cellular network, a wireless network, a wide area network (WAN), a local area network (LAN), or the Internet, among others. An external computing device can connect to the computer system 1100 through the network 1112. In some examples, the external computing device can be an external web server or a cloud computing node.

[0083] It should be understood that the block diagram of Figure 12 is not intended to indicate that computer system 1100 includes all of the components shown in Figure 12. Rather, computer system 1100 can include any suitable fewer or additional components (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.) not shown in Figure 12. Furthermore, the embodiments described herein with respect to computer system 1100 may be implemented with any suitable logic, where logic referred to herein can, in various embodiments, include any suitable hardware (e.g., a processor, an embedded controller, or an application specific integrated circuit, among others), software (e.g., an application, among others), firmware, or any suitable combination of hardware, software, and firmware.

[0084] The present invention may be a system, method, or computer program product, or combination thereof, at any level of technical detail that may be integrated. The computer program product may include one or more computer-readable storage media having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0085] The computer readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or a ridge structure in a groove in which instructions are recorded, or any suitable combination thereof. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0086] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to an individual computing device / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may be comprised of copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing device / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to the individual computing device / processing device for storage in a computer readable storage medium.

[0087] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, such as object-oriented programming languages, such as Smalltalk, C++, and the like, conventional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any kind of network, such as a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0088] Aspects of the present invention are described herein with reference to flowchart illustrations or block diagrams, or combinations thereof, of methods, apparatus (systems), and computer program products or computer programs according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations or block diagrams, or combinations thereof, and combinations of blocks in the flowchart illustrations or block diagrams, or combinations thereof, can be implemented by computer readable program instructions.

[0089] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate means for implementing the functions / operations specified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof, to produce a machine. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer programmable data processing apparatus or other device, or combinations thereof, to function in a particular manner, such that a computer readable storage medium having stored thereon includes an article of manufacture including instructions that implement aspects of the functions / operations specified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof.

[0090] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on a computer, other programmable data processing apparatus, or other device, implement the functions / acts identified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof, to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process.

[0091] The flowchart diagrams and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products or computer programs according to various embodiments of the present invention. In this regard, each block in the flowchart diagrams or block diagrams may represent a module, segment, or portion of an instruction, which includes one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the drawings. For example, two blocks shown in succession may actually be accomplished as one step performed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, depending on the functions involved, or the blocks may be performed in the reverse order. It should be noted that each block of the block diagrams or flowchart diagrams or combinations thereof, as well as combinations of multiple blocks of the block diagrams or flowchart diagrams or combinations thereof, may be implemented by a special purpose hardware-based system that performs the specified functions or operations, or may execute a combination of special purpose hardware and computer instructions.

[0092] The description of various embodiments of the present disclosure is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0093] Various embodiments of the present invention are described herein with reference to the associated drawings. Alternative embodiments of the present invention may be devised without departing from the scope of the present invention. In the description and drawings of this specification, various connection and positional relationships between elements (e.g., above, below, adjacent, etc.) are defined. These connection or positional relationships or combinations thereof may be direct or indirect unless otherwise specified, and the present invention is not intended to be limited in this respect. Thus, connections of entities may refer to either direct or indirect connections, and positional relationships between entities may be direct or indirect positional relationships. Moreover, various tasks and process steps described herein may be incorporated into a more comprehensive procedure or process having additional steps or functionality not described in detail herein.

[0094] The following definitions and abbreviations are to be used for interpretation of the claims and the specification. As used herein, the words "comprise," "includes," "includes," "including," "has," "having," "containing," or "containing," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a composition, mixture, process, method, article, or device that includes a list of elements is not necessarily limited to only those elements, but can include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or device.

[0095] Moreover, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms "at least one" and "one or more" may be understood to include any integer number greater than or equal to one, i.e., 1, 2, 3, 4, etc. The term "plurality" may be understood to include any integer number greater than or equal to two, i.e., 2, 3, 4, 5, etc. The term "connected" may include both an indirect "connected" and a direct "connected."

[0096] The words "about," "substantially," "approximately," and variations thereof are intended to include the degree of error associated with the measurement of a particular quantity based on equipment available at the time of filing. For example, "about" can include a range of ±8%, or ±5%, or ±2% of a given value.

[0097] For the sake of brevity, conventional techniques associated with making and using aspects of the present invention may or may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs for implementing various technical features described herein are well known. Thus, for the sake of brevity, many conventional implementation details are only briefly mentioned herein or omitted entirely without providing details of well-known systems or processes or combinations thereof.

Claims

1. 1. A computer-implemented method for providing interpretable predictions from a machine learning model, comprising: receiving, by a processor of a computing device, a data structure representing a hierarchical structure of a set of features (X) used by one or more predictive models to generate a set of predictions (Y); constructing, by the processor, an interpretability model corresponding to the predictive model by assigning an interpretability to each prediction Yi of the predictive model based on the hierarchical structure, where assigning the interpretability includes: decomposing, by the processor, the set (X) of features of the input data into a number of partitions Xj using the hierarchical structure, where X is represented by the formula: [0010] N is the number of partitions, decomposing, by the processor, each partition from the plurality of partitions into a plurality of subpartitions using the hierarchical structure until an atomic subpartition is obtained; and calculating by said processor an interaction between said sub-partitions by comparing a predicted score for said partition with a predicted score of a sub-partition of said partition; The constructing, outputting, by the processor, an interpretation of the prediction provided by the predictive model based on the interpretability model, where the interpretation represents weights of the one or more features used by the predictive model to generate the prediction. The method comprising:

2. The computer-implemented method of claim 1 , wherein the partitions of features are mutually exclusive subsets of the input data.

3. The computer-implemented method of claim 1 or 2, further comprising generating an overall score for the input data based on the predicted scores for each partition.

4. generating the hierarchical structure, receiving, by the processor, industry data curated by one or more experts; and generating, by the processor, the hierarchical structure using the industrial data by parsing the industrial data into the partitions of features and building a dendrogram model by creating associations between the partitions; The computer-implemented method of any one of claims 1 to 3, further comprising:

5. 5. The computer-implemented method of claim 1, further comprising identifying, by the processor, a recommended action based on the predicted scores for selected features.

6. A computer-implemented method according to any preceding claim, wherein each partition is associated with a corresponding interpretability model.

7. The computer-implemented method of any one of claims 1 to 6, wherein the interpretability model is an attention model.

8. 1. A system comprising: At least one memory storing computer-executable instructions; and At least one processor configured to access the at least one memory and execute a method for providing interpretable predictions from a machine learning model. wherein the method comprises: receiving, by a processor of a computing device, a data structure representing a hierarchical structure of a set of features (X) used by one or more predictive models to generate a set of predictions (Y); constructing, by the processor, an interpretability model corresponding to the predictive model by assigning an interpretability to each prediction Yi of the predictive model based on the hierarchical structure, where assigning the interpretability includes: decomposing, by the processor, the set (X) of features of the input data into a number of partitions Xj using the hierarchical structure, where X is represented by the formula: [0025] N is the number of partitions, decomposing, by the processor, each partition from the plurality of partitions into a plurality of subpartitions using the hierarchical structure until an atomic subpartition is obtained; and calculating by said processor an interaction between said sub-partitions by comparing a predicted score for said partition with a predicted score of a sub-partition of said partition; The constructing, outputting, by the processor, an interpretation of the prediction provided by the predictive model based on the interpretability model, where the interpretation represents a weighting of the one or more features used by the predictive model to generate the prediction. The system comprising:

9. The system of claim 8 , wherein the partitions of features are mutually exclusive subsets of the input data.

10. The system of claim 8 or 9, wherein the method further comprises generating an overall score for the input data based on the predicted scores for each partition.

11. generating the hierarchical structure, receiving industry data curated by one or more experts; and generating the hierarchical structure using the industrial data by parsing the industrial data into the partitions of features and building a dendrogram model by creating associations between the partitions; The system according to any one of claims 8 to 10, further comprising:

12. The system of any one of claims 8 to 11, wherein the method further comprises identifying, by the processor, a recommended action based on the predicted scores for selected features.

13. The system of claim 8 , wherein each partition is associated with a corresponding interpretability model.

14. The system of claim 8 , wherein the interpretability model is an attention model.

15. A computer program comprising: receiving a data structure representing a hierarchical structure of a set of features (X) used by one or more predictive models to generate a set of predictions (Y); constructing an interpretability model corresponding to the predictive model by assigning an interpretability to each prediction Yi of the predictive model based on the hierarchical structure, where assigning the interpretability includes: Decomposing the set of input data features (X) into a number of partitions Xj using the hierarchical structure, where X is expressed as: [0030] N is the number of partitions, decomposing each partition from the plurality of partitions into a plurality of subpartitions using the hierarchical structure until an atomic subpartition is obtained; and calculating interactions between said subpartitions by comparing the predicted score for said partition with the predicted scores of the subpartitions of said partition; The constructing, outputting an interpretation of the prediction provided by the predictive model based on the interpretability model, where the interpretation represents a weighting of the one or more features used by the predictive model to generate the prediction. The computer program causes a processor of a computing device to execute each step of the method comprising:

16. 16. The computer program product of claim 15, wherein the partitions of features are mutually exclusive subsets of the input data.

17. The computer program product of claim 15 , further comprising causing the processor to generate an overall score for the input data based on the predicted scores for each partition.

18. generating the hierarchical structure, receiving industry data curated by one or more experts; and generating the hierarchical structure using the industrial data by parsing the industrial data into the partitions of features and building a dendrogram model by creating associations between the partitions; The computer program product of claim 15 further comprising:

19. The computer program product of claim 15 , wherein each partition is associated with a corresponding interpretability model.

20. The computer program product of claim 15 , wherein the interpretability model is an attention model.

Images