A computer implementation method for constructing decision trees in machine learning, a computer program product including a computer-readable storage medium with program instructions implemented therein, and a system (improvement of the performance of classification and regression trees through dimensionality reduction).

Abstract

To provide a computer-implemented method, computer program product and system, for improving the performance of classification and regression trees through dimensionality reduction.SOLUTION: There is possibly provided a system and method for building and drilling a decision tree for machine learning, in which a drill set is to be received. The decision tree is to be initialized by building a root node, and a root solver is to be drilled with a drill set. A processor can grow the decision tree by repetitively dividing a node of the decision tree. At the node of the decision tree, dimensionality reduction is executed for a feature amount of data of the drill set received at the node, in which dimensionally reduced data is divided based on a routing function, for routing to another node of the decision tree. Dimensionality reduction and division are executed together at the node based on solving a non-linear optimization problem.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] This application generally relates to computers and computer applications, and more particularly, to the creation and training of classification and regression trees using machine learning and dimensionality reduction techniques.

Background Art

[0002] Decision trees are a popular class of machine learning models known for their computational appeal and strong performance in various applications. Decision trees function by learning hierarchical clusters of data generated by recursively splitting the data. While popular, the most basic methods such as classification and regression trees (CART) rely on greedy criteria or heuristics to generate splits and may sacrifice the optimality of the splits to reduce computational cost.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Provided are a computer-implemented method, a computer program product, and a system for improving the performance of classification and regression trees by dimensionality reduction.

Means for Solving the Problems

[0004] The summary of the present disclosure is provided to assist in understanding an end-to-end system of computer systems and methods for enhancing the performance of classification and regression trees using, for example, dimensionality reduction techniques, and is not intended to limit the present disclosure or the invention. It should be understood that the various aspects and features of the present disclosure may advantageously be used separately in some embodiments or in combination with other aspects and features of the present disclosure in other embodiments. Thus, variations and modifications can be made to the computer system or its method of operation or both to achieve different effects.

[0005] A computer implementation method for constructing a decision tree in machine learning may, in one embodiment, include receiving a training set. The method may also include initializing the decision tree by constructing a root node and training a root solver on the training set. The method may grow the decision tree by iteratively splitting the nodes of the decision tree, wherein at each node of the decision tree, dimensionality reduction is performed on the features of the training set data received at the node, the dimensionally reduced data is split based on a routing function for routing to another node of the decision tree, the dimensionality reduction and the splitting are performed together at the node, and the decision tree is grown to include routing nodes and leaf nodes. The method may also include simultaneously training the routing function at the routing node, the solver at the leaf nodes, and the dimensionality reduction at each node of the decision tree by an optimization algorithm.

[0006] In one embodiment, the system may include a processor and a memory device coupled to the processor. The processor may be configured to receive a training set. The processor may be configured to initialize a decision tree by constructing a root node and training a root solver on the training set. The processor may also be configured to grow the decision tree by iteratively splitting the nodes of the decision tree, wherein at each node of the decision tree, dimensionality reduction is performed on the features of the training set data received at the node, the dimensionality-reduced data is split based on a routing function for routing to another node of the decision tree, the dimensionality reduction and the splitting are performed together at the node, and the decision tree is grown to include routing nodes and leaf nodes. The processor may also be configured to simultaneously train the routing function at the routing nodes, the solver at the leaf nodes, and the dimensionality reduction at each node of the decision tree by an optimization algorithm.

[0007] A computer-readable storage medium for storing a program of machine-executable instructions for performing one or more of the methods described herein may also be provided.

[0008] Further features, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings. In the drawings, similar reference numerals indicate identical or functionally similar elements. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows decision tree learning in one embodiment. [Figure 2] Another diagram illustrating dimensionality reduction and decision tree training or learning in one embodiment. [Figure 3] Another diagram illustrating a general decision tree framework in one embodiment. [Figure 4] This is a flowchart illustrating a method in one embodiment of decision tree construction and training. [Figure 5] This figure shows the components of a system in one embodiment that can construct and train decision trees in machine learning. [Figure 6] This is a schematic diagram of an exemplary computer or processing system that can implement a system according to one embodiment. [Figure 7] This figure shows a cloud computing environment in one embodiment. [Figure 8] This figure shows a set of functional abstraction layers provided by a cloud computing environment in one embodiment of the present disclosure. [Modes for carrying out the invention]

[0010] An end-to-end tree learning framework may be provided. In an embodiment, the framework may use nonlinear programming techniques and can operate in data settings where high-dimensional multimodal tabular data may exist that may contain a large number of features and samples. For example, the framework may be a nonlinear programmed decision tree framework. In one embodiment, the framework integrates dimensionality reduction and training processes, for example, integrating dimensionality reduction in the training or learning process of a decision tree. In one embodiment, dimensionality reduction improves the out-of-sample performance of a tree-based supervised learning model. The framework may also implement a regularizer to improve its performance. The framework can be applied, for example, to classification and regression tasks with imbalanced datasets and nonlinear accuracy metrics. The framework can also provide a scalable approach through distributed training. In one embodiment, the framework may employ a method for hierarchical clustering of data. In one embodiment, the branching rules may be based on a probabilistic model, and the framework may recognize downstream learning models. The models can be trained by a scalable distributed-reduced probabilistic gradient algorithm. In one embodiment, the framework may use a structure similar to principal component analysis (PCA) with dimensionality reduction or a nonlinear regularizer within a loop of feature data, or both.

[0011] Decision trees are learning models used in regression and classification. In one or more embodiments, we can present a system or method, or both, that can construct an end-to-end learning scheme that incorporates dimensionality reduction into tree construction. For example, applying or integrating dimensionality reduction into decision tree learning may allow the system or method, or both, to computationally scale optimal classification tree frameworks and regression tree frameworks. By identifying appropriate dimensionality reduction, the performance of decision tree learning can be further improved.

[0012] The decision trees disclosed herein may be implemented, built, and trained on or by one or more computer processors, including, or coupled with, one or more hardware processors. The one or more hardware processors may include components such as, for example, programmable logic devices, microcontrollers, memory devices, or other hardware components, or both, which may be configured to perform the respective tasks described herein. The coupled memory device may be configured to selectively store instructions executable by one or more hardware processors.

[0013] The processor may be a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), other suitable processing components or devices, or one or more combinations thereof. The processor may be coupled with a memory device. The memory device may include random access memory (RAM), read-only memory (ROM), or another memory device, and may store data or processor instructions or both for implementing various functionalities relating to the methods or systems or both described herein. The processor may execute computer instructions stored in memory or received from another computer device or medium.

[0014] TIFF0007872108000001.tif136168

[0015] TIFF0007872108000002.tif67168

[0016] In one embodiment, the non-linear program decision tree framework can be specialized for high-dimensional multi-modal tabular data where many features and samples may exist, can integrate dimensionality reduction and the training process, can be applied to classification tasks and regression tasks with imbalanced datasets and non-linear accuracy metrics, and can form a scalable approach in distributed training.

[0017] FIG. 3 is another diagram illustrating a general decision tree framework in one embodiment. In one aspect, a decision tree can have three components. Router: Routes data left or right r i Node, Transformer: Converts data into a new representation in a low-dimensional space t i Node, Solver: Returns the predicted given data routed to each leaf node s i Node. Routers (e.g., 302, 304) send or split data (e.g., 306) from one node to another. Transformers (e.g., 308, 310, 312, 314, 316) can apply dimensionality reduction to the data. Solvers (e.g., 320, 322, 324) include models fitted to different clusters of data sent to the solver. In one embodiment, each solver can be a different model at the leaf nodes of the decision tree.

[0018] TIFF0007872108000003.tif56167

[0019] TIFF0007872108000004.tif67167

[0020] TIFF0007872108000005.tif26167

[0021] TIFF0007872108000006.tif43167

[0022] TIFF0007872108000007.tif56168

[0023] TIFF0007872108000008.tif56168

[0024] TIFF0007872108000009.tif76168

[0025] TIFF0007872108000010.tif44168

[0026] TIFF0007872108000011.tif45168

[0027] TIFF0007872108000012.tif19167

[0028] TIFF0007872108000013.tif118168

[0029] TIFF0007872108000014.tif106168

[0030] TIFF0007872108000015.tif50167

[0031] TIFF0007872108000016.tif111167

[0032] In one embodiment, during the fine-tuning phase, the framework may globally fit all solver and routing parameters (e.g., at leaf nodes) using stochastic gradient descent (SGD), and the framework may refit the classifier at each leaf node using a basic classifier algorithm. For example, once the growth phase is complete, the framework may globally optimize the tree by solving the problem or equation (1) for the classification setup and the problem or equation (4) for the regression setup. The framework may solve each of these problems using a stochastic gradient descent method such as the Adam algorithm.

[0033] TIFF0007872108000017.tif43167

[0034] In another embodiment, the framework may apply one or more preprocessing steps, unsupervised learning, or both, thereby improving the performance of the learning method. For example, good performance can be achieved by filtering low-variance features, standardizing features, and applying PCA. For example, applying such preprocessing steps can reduce training errors.

[0035] Experiments conducted demonstrate that the methodology of the framework disclosed herein improves training and test accuracy compared to conventional decision tree training techniques that do not include feature reduction. For example, 4020 training samples with 200 features and 4020 test samples were used in the experiment. The number of features in each trial was reduced by a feature reduction factor. It was confirmed that performance improved with fewer features.

[0036] Figure 4 is a flowchart illustrating a method in one embodiment of decision tree construction and training. This method can be implemented or run on one or more computer processors, including, for example, one or more hardware processors. The decision tree learns a sequence of questions, each containing features and split points. In 402, a training set may be received. In 404, the decision tree may be initialized by constructing a root node and training a root solver with the training set. In 406, the decision tree may grow by iteratively splitting its nodes. For example, at a node of the decision tree, dimensionality reduction may be performed on the features of the training set data received at the node, and the data with reduced dimensions may be split based on the optimization of a routing function to route to another node of the decision tree. For example, dimensionality reduction and splitting may be performed together at a node. In 408, the decision tree may be optimized, for example, by fine-tuning. A decision tree can include routing nodes and leaf nodes, and this method may include simultaneously or concurrently performing training on routing functions in the routing nodes, solvers in the leaf nodes, and dimensionality reduction across all nodes of the decision tree using an optimization algorithm.

[0037] In one embodiment, a predetermined topology for a decision tree can be received, where nodes can be iteratively split until the predetermined topology is obtained. In one embodiment, the nodes of the decision tree may include at least routing nodes and leaf nodes, and dimensionality reduction may be performed using optimizations in each of the routing nodes and leaf nodes. In one embodiment, the leaf nodes of the decision tree may include a solver that returns predicted target values. In one embodiment, the leaf nodes of the decision tree may include a regression model that returns predicted target values. In one embodiment, the decision tree may be optimized using a regularizer such as an orthogonal regularizer, a diversified regularizer, or a single routing regularizer or a combination thereof. In one embodiment, the decision tree may be trained to solve a regression problem. In one embodiment, the decision tree may be trained to solve a classification problem. In one embodiment, the training set may include an imbalanced dataset (e.g., not necessarily evenly divided target values), and the model performance metrics may include nonlinear accuracy metrics. For example, nonlinear metrics may include the F1 score, the Matthews correlation coefficient, and the Fowlkes-Mallows index.

[0038] In one embodiment, prediction accuracy can be improved by integrating PCA and decision trees into a unified framework and by using, for example, one or more of the regularizers disclosed herein for end-to-end training. Training time can be reduced by using distributed algorithms to solve problems (1) and (4), such as stochastic gradient descent. In one embodiment, the unified framework can be used for both classification and regression problems.

[0039] Figure 5 shows the components of a system in one embodiment capable of building and training decision trees in machine learning. One or more hardware processors 502, such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA), or a combination thereof, an application-specific integrated circuit (ASIC), or another processor, or both, can be coupled with a memory device 504 and can generate and train a decision tree model based on a training set, and can create predictions or answers to questions based on unseen data. The memory device 504 may include random access memory (RAM), read-only memory (ROM), or another memory device, and may store data or processor instructions or both for implementing various functionalities related to the methods or systems or both described herein. One or more processors 502 may execute computer instructions stored in the memory 504 or received from another computer device or medium. The memory device 504 may store instructions or data or both for operating one or more hardware processors 502, and may also include other programs such as an operating system and instructions or data or both. One or more hardware processors 502 may receive input including a training set. For example, at least one hardware processor 502 may generate a decision tree model in machine learning. In one embodiment, such training data may be stored in a storage device 506 or received from a remote device via a network interface 508 and temporarily loaded into a memory device 504 for building or generating a decision tree model. The trained decision tree model may be stored in the memory device 504 for execution by, for example, one or more hardware processors 502.One or more hardware processors 502 may be coupled with interface devices such as a network interface 508 for communicating with a remote system over a network, and an input / output interface 510 for communicating with input or output devices, or both, such as a keyboard, mouse, display, or other or a combination thereof.

[0040] Figure 6 is a schematic diagram of an exemplary computer or processing system that may implement the system in one embodiment. The computer system is merely an example of a suitable processing system and is not intended to imply any limitations on the scope or functionality of the embodiments of the methodology described herein. The illustrated processing system may operate in a number of other general-purpose or dedicated computing system environments or configurations. Examples of well-known computing systems, environments, or configurations or combinations thereof that may be suitable for use with the processing system shown in Figure 6 include, for example, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable home appliances, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices, and similar.

[0041] A computer system is sometimes described in the general context of a computer system executing computer system executable instructions, such as program modules. Generally, a program module may include routines, programs, objects, components, logic, data structures, etc., that perform a specific task or implement a specific abstract data type. A computer system may also be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked over a communication network. In a distributed cloud computing environment, program modules may reside on both local and remote computer system storage media, including memory storage devices.

[0042] The components of the computer system may include, for example, one or more processors or processing units 12, system memory 16, and a bus 14 that connects various system components, including the system memory 16, to the processor 12. The processor 12 may include a module 30 that performs the methods described herein. The module 30 may be programmed into the integrated circuit of the processor 12, or it may be loaded from memory 16, a storage device 18, or a network 24, or a combination thereof.

[0043] Bus 14 can represent one or more of several types of bus structures, including a memory bus or memory controller, peripheral buses, accelerated graphics ports, and processor or local buses using one of various bus architectures. Examples of such architectures include the Industry Standard Architecture (ISA) bus, Microchannel Architecture (MCA) bus, Expansion ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0044] A computer system may include various computer system-readable media. Such media may be any available media accessible by the computer system and may include both volatile and non-volatile media, and removable and non-removable media.

[0045] The system memory 16 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM), cache memory, or both, or other forms of volatile memory. The computer system may further include other removable / non-removable, volatile / non-volatile computer system storage media. As just one example, the storage system 18 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (e.g., a “hard drive”). Not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media, may also be provided. In such cases, each may be connected to the bus 14 by one or more data medium interfaces.

[0046] The computer system may also communicate with one or more external devices 26 such as a keyboard, pointing device, or display 28, one or more devices that enable a user to interact with the computer system, or any device that enables the computer system to communicate with one or more other computing devices (e.g., a network card, modem, etc.), or a combination thereof. Such communication may occur via an input / output (I / O) interface 20.

[0047] Furthermore, the computer system can communicate with one or more networks 24, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via the network adapter 22. As shown, the network adapter 22 communicates with other components of the computer system via the bus 14. It should be understood that other hardware components, software components, or both, which are not shown, may be used with the computer system. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0048] This disclosure includes a detailed description of cloud computing, but the implementations of the teachings described herein are not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented with any other type of computer environment that is currently known or may be developed in the future. Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or minimal interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four implementation models.

[0049] The characteristics are as follows:

[0050] On-demand self-service: Cloud consumers can unilaterally prepare computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers.

[0051] Broad network access: Computing power is available over the network and accessible through standard mechanisms. This facilitates utilization by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, PDAs).

[0052] Resource pooling: A provider's computing resources are pooled and delivered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated as needed. Generally, consumers have a sense of location independence because they do not manage or know the exact location of the resources provided. However, consumers may be able to identify the location at a higher level of abstraction (e.g., country, state, data center).

[0053] Rapid Elasticity: Computing power can be prepared quickly and flexibly, allowing it to scale out automatically and immediately, and to be quickly released and scale in immediately. To consumers, the computing power available for preparation often appears unlimited and can be purchased in any quantity at any time.

[0054] Measured Services: Cloud systems leverage metric capabilities at a certain level of abstraction, appropriate for the type of service (e.g., storage, processing, bandwidth, active user accounts), to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0055] The service model is as follows:

[0056] Software as a Service (SaaS): The functionality offered to consumers is the ability to use the provider's applications running on a cloud infrastructure. These applications can be accessed from various client devices via thin client interfaces such as web browsers (e.g., webmail). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, except for configuring a limited number of user-specific applications.

[0057] Platform as a Service (PaaS): The functionality offered to consumers is the ability to deploy applications they have created or acquired to cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, and storage, but they can control the deployed applications and, in some cases, the configuration of their hosting environment.

[0058] Infrastructure as a Service (IaaS): The functionality provided to consumers is the provision of processors, storage, networking, and other basic computing resources that enable consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they can control the operating system, storage, and deployed applications, and in some cases, partially control certain network components (e.g., host firewalls).

[0059] The deployment model is as follows:

[0060] Private Cloud: This cloud infrastructure is operated exclusively for a specific organization. This cloud infrastructure can be managed by that organization or a third party and can reside on-premises or off-premises.

[0061] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with common interests (e.g., mission, security requirements, policies, and compliance). This cloud infrastructure can be managed by the organization or a third party and can reside on-premises or off-premises.

[0062] Public Cloud: This cloud infrastructure is provided to a large number of people or large industry groups and is owned by organizations that sell cloud services.

[0063] Hybrid Cloud: This cloud infrastructure combines two or more cloud models (private, community, or public). While maintaining the unique entities of each model, they are bound together by standards or individual technologies to achieve data and application portability (e.g., cloud bursting for load balancing across clouds).

[0064] Cloud computing environments are service-oriented environments that emphasize statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is the infrastructure, which includes a network of interconnected nodes.

[0065] Figure 7 shows a cloud computing environment 50. The cloud computing environment 50 includes one or more cloud computing nodes 10. Local computer devices used by cloud consumers (e.g., PDAs or mobile phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N, or a combination thereof) can communicate with these nodes. The nodes 10 can communicate with each other. The nodes 10 can be grouped physically or virtually (not shown) in one or more networks, such as the private, community, public, or hybrid clouds or a combination thereof. This allows the cloud computing environment 50 to provide infrastructure, platforms, or software as a service, or a combination thereof, and cloud consumers do not need to maintain resources on their local computer devices for these purposes. Note that the types of computer devices 54A-N shown in Figure 7 are merely examples, and it should be understood that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of electronic device via any type of network or network addressable connection (e.g., using a web browser) or both.

[0066] Referring now to Figure 8, a series of functional abstraction layers provided by the cloud computing environment 50 (Figure 7) are shown. It should be understood that the components, layers, and functions shown in Figure 8 are merely illustrative, and the embodiments of the present invention are not limited to these. As illustrated, the following layers and corresponding functions are provided.

[0067] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include a mainframe 61, a reduced instruction set computer (RISC) architecture-based server 62, server 63, blade server 64, storage 65, and a network and network components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0068] The virtualization layer 70 provides an abstraction layer. From this layer, for example, the following virtual entities can be provided: virtual servers 71, virtual storage 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75.

[0069] As an example, the management layer 80 can provide the following functions: Resource preparation 81 enables the dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 82 enables cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. As an example, these resources may include licenses for application software. Security enables not only protection of data and other resources but also identification and verification of cloud consumers and tasks. The user portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 enables the allocation and management of cloud computing resources to ensure that requested service levels are met. Service Level Agreement (SLA) planning and execution 85 enables the pre-arrangement and procurement of cloud computing resources that are expected to be needed in the future in accordance with the SLA.

[0070] Workload layer 90 provides examples of the capabilities available in a cloud computing environment. Examples of workloads and capabilities available from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analysis processing 94, transaction processing 95, and decision tree processing 96.

[0071] The present invention may be a system, method, or computer program product or combination thereof in any executable level of technical detail. The computer program product may include a computer-readable storage medium (or more mediums) having computer-readable program instructions for causing a processor to perform an aspect of the present invention.

[0072] A computer-readable storage medium is a tangible device that holds and stores instructions by an instruction execution device for use. Computer-readable storage media may include, for example, but are not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof as appropriate. More specific examples of computer-readable storage media include portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), writable and erasable random access memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, encoded devices such as punch cards or raised structures with grooves on which mechanical instructions are recorded, and any combination thereof as appropriate. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses passing through fiber optic cables), or electrical signals transmitted over wires.

[0073] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof). The network consists of copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. The network adapter card or network interface of each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on the computer-readable storage medium within each computing / processing device.

[0074] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++ and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions are executable as a standalone software package, either entirely on the user's computer or partially on the user's computer. Alternatively, they may be executable partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by personalizing them using state information of computer-readable program instructions in order to perform aspects of the present invention.

[0075] Aspects of the present invention are described herein with reference to flowcharts or block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and any combination of blocks in a flowchart or block diagram, or both, can be implemented by computer-readable program instructions.

[0076] These computer-readable program instructions can be provided to a computer processor or other programmable data processing device to generate a machine, such that instructions executed via the processor of a computer or other programmable data processing device generate means for implementing functions / operations specified in one or more blocks of a flowchart or block diagram or both. These computer-readable storage media can also be stored in a computer-readable storage medium that can be connected to a computer, a programmable data processing device, or other device or combination of devices that function in a particular way, such that the computer-readable program instructions on which the instructions are stored constitute one of the outputs containing instructions that implement the modes of function / operations specified in one or more blocks of a flowchart or block diagram or both.

[0077] Computer-readable program instructions, like instructions that perform a function / action specified in one or more blocks of a flowchart or block diagram or both on a computer, other programmable device, or other device, can also be loaded onto a computer, other programmable device, or other device and perform a series of operational steps on the computer, other programmable device, or other device to produce a computer-implemented process.

[0078] The flowcharts and block diagrams in the figures illustrate the configuration, functions, and operations of executable embodiments of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or part of an instruction, which constitutes one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions shown in the blocks may differ from the order shown in the figures. For example, two blocks shown consecutively may actually be achieved as a single step, executed simultaneously, substantially simultaneously, partially or entirely in overlapping time, or the blocks may be executed in reverse order depending on their function. It should also be noted that each block in a block diagram or flowchart diagram, or both, and any combination of blocks in a block diagram or flowchart diagram, or both, can be implemented by a special-purpose hardware-based system that performs a specified function or operation, or a combination of special-purpose hardware and computer instructions.

[0079] The terms used herein are for the sole purpose of describing specific embodiments and are not intended to limit the invention. Where used herein, the singular forms “a,” “an,” and “the” are intended to include the plural form unless the context explicitly indicates otherwise. Where used herein, the term “or” is an inclusive operator and may mean “and / or” unless the context explicitly or explicitly indicates otherwise. Where used herein, the terms “comprise,” “comprises,” “comprising,” “include,” “including,” or “having,” or any combination thereof, may specify the presence of a described feature, integer, step, operation, element, or component or combination thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups or combinations thereof. Where used herein, the phrase “in an embodiment” may refer to the same embodiment, but not necessarily the same embodiment. As used herein, the phrase "in one embodiment" may refer to the same embodiment, but not necessarily the same embodiment. As used herein, the phrase "in another embodiment" may refer to a different embodiment, but not necessarily a different embodiment. Furthermore, embodiments, components of embodiments, or both may be freely combined with each other, provided they are not mutually exclusive.

[0080] All corresponding structures, materials, actions, and equivalents or step-plus-function elements in the following claims are intended to include any structures, materials, or actions to perform a function in combination with other claimed elements, as specifically claimed. The description of the invention is presented for illustrative and explanatory purposes, but is not intended to be exhaustive or to limit oneself to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments have been selected and described to best illustrate the principles and practical applications of the invention and to enable those skilled in the art to understand the invention in terms of various embodiments with various modifications suitable for the particular use to be intended.

Claims

1. A computer implementation method for constructing decision trees in machine learning, Receiving the training set, The decision tree is initialized by constructing the root node and training the root solver with the training set, The decision tree is grown by iteratively splitting the nodes of the decision tree, wherein dimensionality reduction is performed on the features of the training set data received at the node, the dimensionality-reduced data is split based on a routing function to route to another node of the decision tree, the dimensionality reduction and the splitting of the data are performed together at the node, and the decision tree is grown to include routing nodes and leaf nodes. The integrated optimization algorithm combines the routing function in the routing node, the solver in the leaf node, and the dimensionality reduction in the routing node and leaf node of the decision tree, thereby simultaneously training the routing function in the routing node, the solver in the leaf node, and the dimensionality reduction in the routing node and leaf node of the decision tree, wherein simultaneous training involves simultaneously obtaining the model parameters of the routing node and the solver of the leaf node, and simultaneously executing the process. Includes, The dimensionality reduction is performed on clusters of data sent to each of the routing nodes, and the first cluster sent to the first routing node of the routing node has a different set of features than the second cluster sent to the second routing node of the routing node. A computer implementation method wherein, in each layer below the root node of the decision tree, a solver is assigned to the trained root solver, and at least two new solvers are trained using a random subset of the training set and fitted as the routing function.

2. The process further includes receiving a predetermined topology for the decision tree, The computer implementation method according to claim 1, wherein the nodes are iteratively divided until the predetermined topology is obtained.

3. The computer implementation method according to claim 1, wherein the leaf node of the decision tree includes the solver that returns a predicted target value.

4. The computer implementation method according to claim 1, wherein the leaf nodes of the decision tree include a regression model that returns a predicted target value.

5. The computer implementation method according to claim 1, further comprising optimizing the decision tree using a regularizer.

6. The computer implementation method according to claim 5, wherein the regularizer includes an orthogonal regularizer.

7. The computer implementation method according to claim 5, wherein the regularizer includes a diversified regularizer.

8. The computer implementation method according to claim 5, wherein the regularizer includes a single routing regularizer.

9. The computer implementation method according to claim 1, wherein the decision tree is trained to solve a regression problem.

10. The computer implementation method according to claim 1, wherein the decision tree is trained to solve a classification problem.

11. The computer implementation method according to claim 1, wherein the training set includes an imbalanced dataset, and the measurement of the model's accuracy performance includes nonlinear metrics.

12. A computer program including program instructions, wherein the program instructions are readable by the device. Receiving the training set, Initializing a decision tree by constructing a root node and training the root solver with the aforementioned training set, The decision tree is grown by iteratively splitting the nodes of the decision tree, wherein dimensionality reduction is performed on the features of the training set data received at the node, the dimensionality-reduced data is split based on a routing function to route to another node of the decision tree, the dimensionality reduction and the splitting of the data are performed together at the node, and the decision tree is grown to include routing nodes and leaf nodes. The integrated optimization algorithm combines the routing function in the routing node, the solver in the leaf node, and the dimensionality reduction in the routing node and leaf node of the decision tree, thereby simultaneously training the routing function in the routing node, the solver in the leaf node, and the dimensionality reduction in the routing node and leaf node of the decision tree, wherein simultaneous training involves simultaneously obtaining the model parameters of the routing node and the solver of the leaf node, and simultaneously executing the process. The device is made to perform the following: The dimensionality reduction is performed on clusters of data sent to each of the routing nodes, and the first cluster sent to the first routing node of the routing node has a different set of features than the second cluster sent to the second routing node of the routing node. A computer program in which, in each layer below the root node of the decision tree, a solver is assigned to the trained root solver, and at least two new solvers are trained using a random subset of the training set and fitted as the routing function.

13. The computer program according to claim 12, wherein the nodes are iteratively divided until a predetermined topology is obtained.

14. The computer program according to claim 12, wherein the leaf node of the decision tree includes the solver that returns a predicted target value.

15. The computer program according to claim 12, wherein the leaf nodes of the decision tree include a regression model that returns a predicted target value.

16. The device is a computer program according to claim 12, which optimizes the decision tree using a regularizer.

17. The computer program according to claim 16, wherein the regularizer includes at least one of an orthogonal regularizer, a diversified regularizer, and a single-routing regularizer.

18. Processor and A memory device coupled to the aforementioned processor, Includes, The aforementioned processor includes at least, Receiving the training set, Initializing a decision tree by constructing a root node and training the root solver with the aforementioned training set, The decision tree is grown by iteratively splitting the nodes of the decision tree, wherein dimensionality reduction is performed on the features of the training set data received at the node, the dimensionality-reduced data is split based on a routing function to route to another node of the decision tree, the dimensionality reduction and the splitting of the data are performed together at the node, and the decision tree is grown to include routing nodes and leaf nodes. The integrated optimization algorithm combines the routing function in the routing node, the solver in the leaf node, and the dimensionality reduction in the routing node and leaf node of the decision tree, thereby simultaneously training the routing function in the routing node, the solver in the leaf node, and the dimensionality reduction in the routing node and leaf node of the decision tree, wherein simultaneous training involves simultaneously obtaining the model parameters of the routing node and the solver of the leaf node, and simultaneously executing the process. It is configured to do the following: The dimensionality reduction is performed on clusters of data sent to each of the routing nodes, and the first cluster sent to the first routing node of the routing node has a different set of features than the second cluster sent to the second routing node of the routing node. A system in which, in each layer below the root node of the decision tree, a solver is assigned to the trained root solver, and at least two new solvers are trained using a random subset of the training set and fitted as the routing function.

19. The system according to claim 18, wherein the leaf nodes of the decision tree include a regression model that returns a predicted target value.

Images