Optimal Interpretable Decision Trees Using Integer Programming Techniques

Abstract

Aspects of the present invention include optimal interpretable decision trees using integer linear programming techniques. A non-limiting example computer-implemented method includes using a processor to receive multiple data inputs from a process and using the processor to select a data subset from the multiple data inputs by solving a linear programming problem to obtain a solution. The method uses the processor to construct and optimize an optimal decision tree based on the data subset, and uses the processor to alert a user when a prediction of the optimal decision tree is greater than a threshold value.

Description

[Technical field]

[0001] The present invention relates generally to predictive modeling, and more specifically to optimal interpretable decision trees using integer linear programming techniques. [Background technology]

[0002] Predictive models are useful in process control environments. For example, blast furnaces, oil refineries, and supply chain management all rely on process control. Deep Neural Networks ("DNN") and Optimal Decision Trees ("ODT") can be used for process control. DNNs are artificial neural networks with multiple layers between the input and the output. DNNs find the correct mathematical operations to turn the input into an output. ODTs are decision trees that have an optimal tradeoff between training accuracy and model simplicity.

[0003] Decision tree based learners need to be simple and accurate. Thus, there is a need in the industry to address the above-mentioned problems. Summary of the Invention

[0004] Viewed from a further aspect, the present invention provides a computer program product for predictive modelling, the computer program product comprising a computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit to perform a method performing steps of the present invention.

[0005] Viewed from a further aspect, the present invention provides a computer program stored on a computer readable medium and loadable into an internal memory of a digital computer, the computer program comprising software code portions for performing the steps of the present invention when said program is run on a computer.

[0006] Viewed from a further aspect, the present invention provides a computer program product for generating and optimizing decision trees, the computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by the computer to cause the computer to perform a method including receiving a plurality of data inputs from a process; selecting a data subset from the plurality of data inputs by solving a linear programming problem to obtain a solution; constructing and optimizing an optimal decision tree ("ODT") based on the data subset; and alerting a user when a prediction of the optimal decision tree is greater than a threshold value.

[0007] An embodiment of the present invention is directed to optimal interpretable decision trees using integer linear programming techniques. A non-limiting example computer-implemented method includes using a processor to receive a plurality of data inputs from a process, and using the processor to select a data subset from the plurality of data inputs by solving a linear programming problem to obtain a solution. The method uses the processor to construct and optimize an optimal decision tree based on the data subset, and uses the processor to alert a user when a prediction of the optimal decision tree is greater than a threshold value.

[0008] Other embodiments of the present invention embody features of the above methods in computer systems and computer program products.

[0009] Further technical features and advantages are realized by the techniques of the present invention. Embodiments and aspects of the present invention are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, reference should be made to the detailed description and drawings.

[0010] The particulars of the exclusive rights set forth herein are particularly pointed out and distinctly claimed in the claims at the end of the specification. The foregoing and other features and advantages of embodiments of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0011] [Figure 1] 1 is a flow diagram of a process for implementing an optimal interpretable decision tree according to an embodiment of the present invention. [Diagram 2] 4 is a flow diagram of a data selection process according to an embodiment of the present invention. [Diagram 3] FIG. 1 illustrates data set selection according to one embodiment of the present invention. [Figure 4] FIG. 1 illustrates a cloud computing environment according to one embodiment of the present invention. [Diagram 5] FIG. 2 illustrates abstraction model layers according to one embodiment of the present invention. [Figure 6] FIG. 1 illustrates details of an exemplary computing system in which aspects of the present invention may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The diagrams shown herein are exemplary. There may be many variations to the diagrams or operations described herein without departing from the scope of the invention. For example, operations may be performed in a different order, or operations may be added, deleted, or modified. Also, the term "coupled" and variations thereof describe having a communication path between two elements and do not imply a direct connection between elements without an intervening element / connection between them. All of these variations are considered part of this specification.

[0013] One or more embodiments of the present invention provide a robust decision tree-based learner designed to learn provably optimal two or three deep trees using integer programming techniques. This provides both interpretability and easy optimization. A global solution for the ODT that is scalable is determined by using mixed integer linear programming. An embodiment of the present invention uses data selection of the training data a priori without loss of generalization by choosing a subset of data points that captures as much information about the entire data set as possible, and then a mixed integer programming ("MIP") formulation for the ODT is established uniquely for this data subset. In particular, a linear programming ("LP") based subset selection procedure is implemented. Practical applications such as continuous process fault prediction are achieved by implementing embodiments of the present invention.

[0014] Mathematical optimization techniques have always been the primary tool for solving machine learning ("ML") problems, but MIPs have rarely been explored in the past, mainly due to their poor tractability. However, developments in MIP theory, optimization solvers, coupled with improvements in hardware, have led to a staggering 800 billion times speedup in MIPs compared to 25 years ago. Thus, ML has begun to adopt MIP-based models, and many methods have been proposed to leverage mixed integer programming to address classical ML tasks where heuristics have traditionally been used. Among attempts to study the interplay between MIPs and ML, the construction of ODTs is among the most popular, mainly due to the combinatorial structure of decision trees and their interpretability. Many related MIP formulations have been proposed to train ODTs for different purposes.

[0015] Previous attempts at solutions have issues with interpretability and ease of optimization. For example, deep neural networks are essentially black boxes with little to no insight into how the network works internally. Highly nonlinear models are difficult to optimize; in other words, it is hard to find local minima. Solutions involving deep decision trees used in the past are difficult and time consuming to optimize. They too can suffer from interpretability issues.

[0016] One or more embodiments of the present invention address one or more of the above shortcomings of the prior art by providing a technical solution that includes a simple decision tree that first performs data selection of the training data. The subset is selected using a linear programming based data selection method. This subset of data is used in a MIP formulation for the ODT. This results in performance enhancements in interpretability (the degree to which hum can understand the reason for the decision), tractability (the ability to return an output within reasonable time limits corresponding to the data size), training accuracy, and generalization ability or testing accuracy.

[0017] Turning now to Figure 1, a flow diagram of the process implemented by an optimal interpretable decision tree is generally shown, in accordance with one or more embodiments of the present invention. At block 110, data selection is performed. An LP-based data selection process is implemented to act as a geometric approximation of the point set. The process assumes an optimal classification tree with very high training accuracy, where several data points in I0 with label y are all correctly classified and assigned to the same leaf node. At block 110, the process selects I0 as a subset of the training set, x i If we use the training samples with I0 removed, then by correctly classifying the data points in I0, we will automatically correctly classify the points inside the convex hull, so I0conv({x i All data points with the same label y inside the convex hull of |i∈I0) are removed from the training set. The number of data points inside the convex hull is maximized in some embodiments of the present invention.

[0018] Since there is no information about the optimal classification tree a priori, here the process uses a trained decision tree from some other heuristic method as a guide and then does data selection within each leaf node. Then among those data points within each leaf node, the process does data selection separately. Finally, the process combines all the selected data subsets from each leaf node and trains a classification tree on this dataset using MIP-ODT.

[0019] In general, for data selection, the process is as follows: (1) conv({x i Select a data subset I0 according to two criteria: (1) the number of points inside |i∈I0) is as many as possible or maximized, and (2) the number of selected sample points |I0| is as few as possible. For example, the remaining sample points for training |I0| are less than 10% of the original dataset, while the number of samples inside the convex hull of I0 is more than 90%. An example is given in Figure 3.

[0020] The basic binary LP for data selection is as follows: For each cluster N ⊆ [n],

[0021]

number

[0022] where f, g are two parameter vectors with non-negative components, j indicates whether data point j is selected or not, and b j indicates whether j can be expressed as a convex combination of selected data points, and λ ji denotes the convex combination coefficient of point i when we represent point j as a convex combination of several other points.

[0023] b j When λ = 1, the first two constraints are ji >0 and point x jLet us express approximately as a convex combination of other points i, and b j When = 0, these two constraints then satisfy λ for all i. ji = 0, so the third inequality 0 ≦ λ holds. ji ≦a i means that the process can use the selected data point to represent other data points. i = 1. The last constraint a j +b j ≦1 states that any selected data point cannot be expressed as a convex combination of other selected data points. Depending on the choice of f, g and the convex combination constraints, there will generally be many different variations on the above formula.

[0024] When f=0 and g=1, the process can project out a and obtain the following simpler binary LP:

[0025]

number

[0026] The next result shows that this binary LP is the following LP:

[0027]

number

[0028] For any optimal solution (b, λ) of the above equation, there exists λ such that (b, λ) is an optimal solution of the previous equation, and vice versa.

[0029] For some pre-given threshold parameters β1, β2 ∈ (0,1), the data selection process (block 110) is given in FIG. 2, which will be explained later.

[0030] After data selection, a decision tree is constructed and optimized (block 120) as follows. Without loss of generality, the process assumes that the classes of the dataset are [Y]:={1, 2, ..., Y}, where Y is the number of classes. Each data point i ∈ [n] is denoted by (x i ,y i ), where n is the number of data points and x i is a d-dimensional vector, and y i ∈{1,...,Y}. The process is F q Let ⊆[d] be the index set of all numerical features, while F c denotes the index set of all features that belong to the classification. In this formulation, the process q = [n]). Even though the trained ODT may be unbalanced in practice, the formulation is established for a balanced binary tree with depth D, where the branch nodes of the tree are B:=[2 D -1], and the leaf nodes are D ,...,2 D+1 The process is denoted as A R (l) and A L We also use (l) to denote the set of ancestors of a leaf node l that the left (right) branch follows on the path from the root node to the leaf node l.

[0031] When training decision trees with MIP, it is almost always preferable to train multivariate decision trees rather than classical single-variable decision trees, since the latter require additional binary variables that are forced at each branch node until a feature can be used for splitting. Like the generalization to single-variable decision trees, multivariate decision trees are more difficult to train with general heuristic algorithms. The only difference between these two types of decision trees is whether the branching hyperplane must be parallel to the axis or not. Since multivariate decision trees are easier to train with MIP models, the process will only be described in terms of optimal multivariate decision trees, even though single-variable decision trees can also be formulated by adding one more constraint to the formulation for each branch node. The MIP-ODT formulation is

[0032]

number

[0033] A balanced binary tree with depth D can be characterized by only two elements: the hyperplanes branching at each branch node, and the labels assigned to each leaf node. In our MIP-ODT, the branching hyperplane at branch node b is <h b ,x i >=g b and the branching law is given by: If data i is <h b ,x i >≦g b If l is the left branch of u, then it goes to the right branch. If l is the right branch of u, then it goes to the left branch of u. l By assuming that there exists a ∈[Y], u l It is shown that:

[0034] Binary variables are

number

number

number

number

number

[0035] ci and

number

number

number

[0036] Constraints (1f), (1g), and (1i) are about formulating a branching rule at each node b ∈ B such that if data i goes to the left branch at node b, then

number

number

number

number

number

number

number

[0037] This is not exactly the same as our branching rule; when data i goes to the right branch, it instead says

number

number

[0038] The soft margin variable m i,bPenalizing is motivated by Support Vector Machines ("SVM"), where the process wants the diverging hyperplane to not be too close to all data points. This gives us the objective function:

number

[0039] By Occam's razor, the process is bounded by an extra penalty term. Σ b ||h b ||0 We also take into account the simplicity (sparseness) of the model by adding

[0040] Adding the misclassification numbers to the objective yield gives

number

[0041]

number

number

[0042] A series of cutting planes may be used to further enhance the above MIP-ODT formulation. The cutting planes are: ∀l∈L,Σ i c i ≧Σ i e i,l -s Y ∀l∈L,Σ i (c i +e i,l ) ≧ s1(Y-2 D+1), and s1+···+s Y-2 D ≦Σ i c i ≦ns Y Includes.

[0043] Here, the process is: for any k ∈ [Y], N k :={i∈[n]:x i,d =k}, S:={|N k |:k∈[Y]}, and |N1|=s1≦|N2|=s2≦...≦|N Y |=s Y Assume that:

[0044] After generation and optimization of the ODT, a check is made to determine if the prediction resulting from the ODT is below a threshold value (block 130). If so, the assets in the process are under control (block 150). If not, the process alerts the user and recommends control actions based on the analysis of the ODT (block 140). The control actions are based on the feature vector input x i This is done by adjusting the value of i This results in changes in

[0045] The process described with respect to FIG. 1 may be used, for example, for continuous process fault prediction, which is a type of supervised anomaly prediction. It is designed to alert the user when a process fault is detected. It is informed based on historical process data and serves as ground truth for training an AI-based model. Various rule types, such as threshold, trend, or multiple, may be implemented. For example, a threshold may alert, for example, when a tank volume is below a minimum amount. A trend may alert, for example, when there is a decrease in throughput or when there is a power drop. A multiple may be triggered when there is a loss of flow without a set point, or, for example, when the temperature is below a threshold and a rate above a second threshold is predicted.

[0046] For process fault prediction, an ODT is generated with the goal of predicting process faults. A sequence of process sensors and labeled upsets are transformed into a classification problem. Fault events are extracted using look-ahead windows to produce training targets that enable a learning model to anticipate the events. Features are extracted from one or more look-back windows using a summary function.

[0047] FIG. 2 shows a flow diagram of a data selection process according to an embodiment of the present invention. If β1, β2 ∈ (0,1), then

[0048]

number

[0049] I N :={i∈N:b i = 1}, λ = T(λ), J N :={i∈N:∃j∈I N , where λ ji ≧1 / (d+1)}, K N :=N\(I N UJ N In block 220, |I N If | / |N|≧1−β1, select N / IN as the training set (block 230). N If | ≥ β2 | N | ≥ J N As the training set (block 250), otherwise, in block 260, N For the first β2|N|-|J N Let the points in | be the selected training set. N Choose together.

[0050] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0051] The characteristics are as follows: On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without the need for human interaction with the service provider. Broad network access: Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (eg, mobile phones, laptops, and PDAs). Resource Sharing: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. There is a notion of location independence, in that consumers generally have no control or knowledge of the exact location of a given resource, but may be able to specify location at a higher level of abstraction (e.g., country, state, or data center). Rapid elasticity: Capability can be provisioned quickly and elastically, sometimes automatically, to quickly scale out, and can be de-provisioned quickly to quickly scale in. To the consumer, the capacity available for provisioning often appears to be unlimited and can be purchased in any quantity at any time. Measured service: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage may be monitored, controlled, and reported, providing transparency to both providers and consumers of utilized services.

[0052] The service modules are: Software as a Service (SaaS): The functionality given to the consumer is the use of a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even the individual application functions, with the possible exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The functionality offered to the consumer is the deployment onto a cloud infrastructure of consumer-generated or acquired applications that are created using programming languages ​​and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but has control over the developed applications and possibly the application hosting environment configuration. Infrastructure as a Service (IaaS): The functionality offered to the consumer is to provision the processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and in some cases, limited control over select networking components (e.g., host firewalls).

[0053] The deployment model is as follows: Private Cloud: The cloud infrastructure is operated solely for one organization. It may be managed by that organization or a third party and can exist on-premises or off-premises. Community Cloud: The cloud infrastructure is shared by several organizations to support a specific community with shared concerns (e.g., mission, security requirements, policy, and compliance considerations). It may be managed by the organization or a third party and may exist on-premises or off-premises. Public Cloud: The cloud infrastructure is made available to the general public or large corporate groups and is owned by an organization that sells cloud services. Hybrid Cloud: A cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are tied together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0054] A cloud computing environment is a service oriented approach with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0055] Referring now to FIG. 4, an exemplary cloud computing environment 50 is illustrated. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10, with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, or a vehicle computer system 54N, or combinations thereof, can communicate. The nodes 10 can communicate with each other. They may be physically or virtually grouped in one or more networks, such as a private, community, public, or hybrid cloud, or combinations thereof, as described above (not shown). This enables the cloud computing environment 50 to provide an infrastructure, platform, or software-as-a-service, or combinations thereof, for which the cloud consumers do not need to maintain resources on their local computing devices. It will be understood that the types of computing devices 54A-N shown in FIG. 4 are intended as examples only, and that the computing node 10 and cloud computing environment 50 may communicate with any type of computerized device via any type of network, or a network-addressable connection (e.g., using a web browser), or both.

[0056] Referring now to Figure 5, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 4) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 5 are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0057] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include mainframes 61, RISC (reduced instruction set computing) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67, and database software 68.

[0058] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0059] In one example, the management layer 80 may provide the functions described below: Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity authentication for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides cloud computing resource allocation and management such that requested service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-configuration and procurement for cloud computing resources where future demand is predicted according to SLAs.

[0060] The workload tier 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and ODT construction and optimization processing 96.

[0061] FIG. 6 illustrates details of an exemplary computing system that may implement aspects of the present invention. FIG. 6 illustrates a high-level block diagram of a computer system 600 that may be used to implement one or more aspects of the present invention. The computer system 600 may act as a media device and may implement the present invention in its entirety or in cooperation with other computers and cloud-based systems to implement the present invention. More specifically, the computer system 600 may be used to implement some hardware components of embodiments of the present invention. Although one exemplary computer system 600 is shown, the computer system 600 includes a communication path 655 that connects the computer system 600 to additional systems (not shown), which may include one or more wide area networks (WANs), such as the Internet, an intranet, or a wireless communication network, or a combination thereof, or a local area network (LAN). The computer system 600 and the additional systems communicate over the communication path 655, for example, to communicate data between them.

[0062] Computer system 600 includes one or more processors, e.g., processor 605. Processor 605 is connected to a communications infrastructure 660 (e.g., a communications bus, a cross-over bar, or a network). Computer system 600 may include a display interface 615 that transmits graphics, text, and other data from communications infrastructure 660 (or from a frame buffer, not shown) for display on a display unit 625. Computer system 600 may include a main memory 610, preferably a random access memory (RAM), and may also include a secondary memory 665. Secondary memory 665 may include, e.g., a hard disk drive 620, representing, e.g., a floppy disk drive, a magnetic tape drive, or an optical disk drive, or a removable storage drive 630, or both. The removable storage drive 630 reads from and / or writes to the removable storage unit 640 in a manner well known to those skilled in the art. The removable storage unit 640 represents, for example, a floppy disk, compact disk, magnetic tape, or optical disk that may be read from and / or written to by the removable storage drive 630. As will be appreciated, the removable storage unit 640 includes a computer readable medium having computer software and / or data stored therein.

[0063] In alternative embodiments, the secondary memory 665 may include other similar means that allow computer programs or other instructions to be loaded into the computer system. Such means may include, for example, a removable storage unit 645 and an interface 635. Examples of such means may include program packages and package interfaces (e.g., those found in video game devices), removable memory chips (e.g., EPROM, or PROM) and associated sockets, and other removable storage units 640 and interfaces 635 that allow software and data to be transferred from the removable storage unit 645 to the computer system 600. In addition, the camera 670 is in communication with the processor 605, the main memory 610, and other peripherals and storage through the communication interface 660.

[0064] Computer system 600 may also include a communications interface 650. Communications interface 650 allows software and data to be transferred between the computer system and external devices. Examples of communications interface 650 may include a modem, a network interface (e.g., an Ethernet card), a communications port, or a PCM-CIA slot and card, etc. The software and data transferred through communications interface 650 are in the form of signals, which may be, for example, electronic, electromagnetic, optical, or other signals that can be received by communications interface 650. These signals are provided to communications interface 650 through communications paths (i.e., channels) 655. Communications paths 655 carry signals and may be implemented using wire or cable, optical fibers, telephone lines, cellular phone links, RF links, or other communications channels, or combinations thereof.

[0065] In this description, the terms "computer program medium," "computer usable medium," and "computer readable medium" are used to generally refer to media such as the main memory 610 and the secondary memory 665, the removable storage drive 630, and the hard disk installed in the hard disk drive 620. Computer programs (also called computer control logic) are stored in the main memory 610, or the secondary memory 665, or both. Computer programs may also be received via the communications interface 650. Such computer programs, when executed, enable the computer system to perform the features of the present disclosure as described herein. In particular, the computer programs, when executed, enable the processor 605 to perform the features of the computer system. Thus, such computer programs represent the controller of the computer system.

[0066] Many of the functional units described herein are termed modules. Embodiments of the present invention apply to a wide variety of module implementations. For example, a module may be implemented as hardware circuitry comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, or the like.

[0067] Modules may also be implemented in software for execution by various types of processors. An identified module of executable code may comprise one or more physical or logical blocks of computer instructions, which may be organized as, for example, an object, procedure, or function. Nevertheless, an executable file for an identified module need not be physically located together, but may comprise heterogeneous instructions stored in different locations that, when logically coupled together, comprise the module and achieve the stated purpose of the module.

[0068] The present invention may be a system, method, and / or computer program product at any possible level of technical detail of integration. The computer program product may include one or more computer-readable storage media having computer-readable program instructions thereon that cause a processor to implement aspects of the present invention.

[0069] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or raised structures in grooves in which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse traveling through a fiber optic cable), or an electrical signal transmitted over a wire.

[0070] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and sends the computer readable program instructions for storage in a computer readable storage medium in the respective computing / processing device.

[0071] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages, e.g., Smalltalk®, C++®, etc., and procedural programming languages, e.g., the “C” programming language® or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0072] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0073] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, produce means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium as capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0074] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a sequence of operational steps to be executed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0075] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur in an order other than that shown. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, may be realized by a dedicated hardware-based system that performs the specified functions or operations or implements a combination of dedicated hardware and computer instructions.

[0076] The description of various embodiments of the present invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used in this specification has been selected to best explain the principles of the embodiments, practical applications, or technical improvements over the art found in the market, or to enable those skilled in the art to understand the embodiments described herein.

Claims

1. 1. A computer-implemented method for predictive modeling, comprising: receiving a plurality of data inputs from a process using a processor; performing data selection with said processor by solving a linear programming problem among the data points in each leaf node to obtain a solution using a decision tree trained from other heuristic methods as a guide, and selecting data subsets from said plurality of data inputs by combining all data subsets selected from each leaf node, wherein all data points with the same label that are inside a convex hull of said selected data subsets are removed from a training set; constructing, using the processor, an optimal decision tree based on the selected data subset; using the processor to alert a user when the optimal decision tree prediction is greater than a threshold value; The method includes:

2. 1. A computer-implemented method for predictive modeling, comprising: receiving a plurality of data inputs from a process using a processor; performing data selection with said processor by solving a linear programming problem among the data points in each leaf node to obtain a solution using a decision tree trained from other heuristic methods as a guide, and selecting data subsets from said plurality of data inputs by combining all data subsets selected from each leaf node, wherein all data points with the same label that are inside a convex hull of said selected data subsets are removed from a training set; constructing, using the processor, an optimal decision tree based on the selected data subset; using the processor to alert a user when the optimal decision tree prediction is greater than a threshold value; performing, with the processor, data selection within each leaf node of the ODT to select the data subset; The method includes:

3. 1. A computer-implemented method for predictive modeling, comprising: receiving a plurality of data inputs from a process using a processor; performing data selection with said processor by solving a linear programming problem among the data points in each leaf node to obtain a solution using a decision tree trained from other heuristic methods as a guide, and selecting data subsets from said plurality of data inputs by combining all data subsets selected from each leaf node, wherein all data points with the same label that are inside a convex hull of said selected data subsets are removed from a training set; constructing, using the processor, an optimal decision tree based on the selected data subset; using the processor to alert a user when the optimal decision tree prediction is greater than a threshold value; using the set of cut planes to enhance the ODT with the processor; The method includes:

4. 1. A computer-implemented method for predictive modeling, comprising: receiving a plurality of data inputs from a process using a processor; performing data selection with said processor by solving a linear programming problem among the data points in each leaf node to obtain a solution using a decision tree trained from other heuristic methods as a guide, and selecting data subsets from said plurality of data inputs by combining all data subsets selected from each leaf node, wherein all data points with the same label that are inside a convex hull of said selected data subsets are removed from a training set; constructing, using the processor, an optimal decision tree based on the selected data subset; using the processor to alert a user when the optimal decision tree prediction is greater than a threshold value; Selecting a data subset from the plurality of data inputs with the processor includes selecting the data subset such that the number of points inside a convex hull of the data subset is maximized; The method includes:

5. 5. The computer-implemented method of claim 1, further comprising: using the processor to recommend a control action when the prediction of the optimal decision tree is greater than the threshold value.

6. 5. The computer-implemented method of claim 1, further comprising: using the processor to perform data selection within each leaf node of the ODT to select the data subset.

7. The computer-implemented method of claim 1, 2, 4 or 6, further comprising, with the processor, using a set of cutting planes to enhance the ODT.

8. 8. The computer implemented method of claim 1, wherein using the processor to select a data subset from the plurality of data inputs comprises selecting the data subset such that the points inside a convex hull of the data subset are maximized.

9. The computer-implemented method of any one of claims 1 to 8, wherein the ODT comprises a multivariate decision tree.

10. In response to alerting the user, using the processor to perform a control action based on an analysis of the optimal decision tree. The computer-implemented method of any one of claims 1 to 9, further comprising:

11. 1. A system for predictive modeling, comprising: Memory, A processor communicatively coupled to the memory and operable to execute instructions stored in the memory, the instructions causing the processor to: Accept multiple data inputs from the process, using a decision tree trained from other heuristic methods as a guide, performing data selection by solving a linear programming problem among the data points in each leaf node to obtain a solution, and selecting a data subset from the multiple data inputs by combining all data subsets selected from each leaf node, where all data points with the same label that are inside the convex hull of the selected data subsets are removed from the training set; constructing an optimal decision tree (hereinafter, the “optimal decision tree” is referred to as “ODT”) based on the selected data subset; alerting a user when the optimal decision tree prediction is greater than a threshold value; A system comprising the processor.

12. 1. A system for predictive modeling, comprising: Memory, A processor communicatively coupled to the memory and operable to execute instructions stored in the memory, the instructions causing the processor to: Accept multiple data inputs from the process, using a decision tree trained from other heuristic methods as a guide, performing data selection by solving a linear programming problem among the data points in each leaf node to obtain a solution, and selecting a data subset from the multiple data inputs by combining all data subsets selected from each leaf node, where all data points with the same label that are inside the convex hull of the selected data subsets are removed from the training set; constructing an optimal decision tree (hereinafter, the “optimal decision tree” is referred to as “ODT”) based on the selected data subset; alerting a user when the optimal decision tree prediction is greater than a threshold value; The processor, The instructions further cause the processor to perform data selection within each leaf node of the ODT to select the data subset. system.

13. 1. A system for predictive modeling, comprising: Memory, A processor communicatively coupled to the memory and operable to execute instructions stored in the memory, the instructions causing the processor to: Accept multiple data inputs from the process, using a decision tree trained from other heuristic methods as a guide, performing data selection by solving a linear programming problem among the data points in each leaf node to obtain a solution, and selecting a data subset from the multiple data inputs by combining all data subsets selected from each leaf node, where all data points with the same label that are inside the convex hull of the selected data subsets are removed from the training set; constructing an optimal decision tree (hereinafter, the “optimal decision tree” is referred to as “ODT”) based on the selected data subset; alerting a user when the optimal decision tree prediction is greater than a threshold value; The processor, The instructions further cause the processor to use a set of cutting planes to enhance the ODT. system.

14. 1. A system for predictive modeling, comprising: Memory, A processor communicatively coupled to the memory and operable to execute instructions stored in the memory, the instructions causing the processor to: Accept multiple data inputs from the process, using a decision tree trained from other heuristic methods as a guide, performing data selection by solving a linear programming problem among the data points in each leaf node to obtain a solution, and selecting a data subset from the multiple data inputs by combining all data subsets selected from each leaf node, where all data points with the same label that are inside the convex hull of the selected data subsets are removed from the training set; constructing an optimal decision tree (hereinafter, the “optimal decision tree” is referred to as “ODT”) based on the selected data subset; alerting a user when the optimal decision tree prediction is greater than a threshold value; The processor, selecting a data subset from the plurality of data inputs includes selecting the data subset such that a number of points inside a convex hull of the data subset is maximized; system.

15. The system of any one of claims 11 to 14, wherein the instructions further cause the processor to recommend a control action when the prediction of the optimal decision tree is greater than the threshold value.

16. The system of any one of claims 11 or 13-14, wherein the instructions further cause the processor to perform data selection within each leaf node of the ODT to select the data subset.

17. The system of claim 11-12, 14 or 16, wherein the instructions further cause the processor to use a set of cutting planes to enhance the ODT.

18. 18. The system of claim 11, 12 or 17, wherein selecting a data subset from the plurality of data inputs comprises selecting the data subset such that the points inside a convex hull of the data subset are maximized.

19. The system of any one of claims 11 to 18, wherein the ODT comprises a multivariate decision tree.

20. The instructions cause the processor to, in response to alerting the user, use the processor to perform a control action based on analysis of the optimal decision tree. A system according to any one of claims 11 to 19.

21. A computer program for causing a computer to carry out the steps of the method according to any one of claims 1 to 10.

22. A computer-readable medium having a computer program recorded thereon, the computer program being for causing a computer to execute each step of the method according to any one of claims 1 to 10.

Images