Deep Forest model development and training

Abstract

Automatic development and training of deep forest models for analyzing data includes growing a random forest of decision trees using the data, determining the out-of-bag (OOB) predictions of the forest, appending the OOB predictions to the dataset, and growing additional forests using the dataset including the appended OOB predictions, and merging the outputs of the additional forests, and then using the model to classify data outside the training dataset.

Description

Background Art

[0001] The present disclosure generally relates to systems and methods for developing and training models for analyzing data. In particular, the present disclosure relates to systems for automatically developing and training models for analyzing data using deep random forests.

[0002] Deep learning is well-known, at least in part, due to the successful application of deep neural networks, particularly in tasks involving images and speech. However, deep neural networks also have drawbacks. They can require large amounts of data for training, making them less suitable for small-scale data. Deep neural networks can also be very complex models with numerous hyperparameters. These parameters may require careful tuning to achieve the desired learning accuracy, making training deep neural networks more of an art than a science or engineering endeavor.

[0003] Zhou et al. proposed a new deep learning method, gcForest (multi-granularity cascade forest) [Z.-H. Zhou and F. Ji. Deep forest: towards an alternative to deep neural networks. arXiv:1702.08835v2, 2017]. This method attempts to implement a key deep learning concept, representation learning, through the cascade structure of random forests, where each layer of the cascade receives feature information created by the previous layer and outputs its results to the next layer. gcForest can achieve performance competitive with deep neural networks, but with fewer of the aforementioned drawbacks.

[0004] Random forest was proposed by Breiman in 2001 [L. Breiman. Random forests. Machine Learning, 45(1): 5-32, 2001]. It is a collection of decision trees. When growing these decision trees in the collection, special processing is applied. First, each tree is grown on a bootstrap sample. The bootstrap sample is obtained by sampling with replacement from the training data, and the sample size is equal to the size of the training data. Second, each tree needs to be fully grown on the sample until each leaf node contains only instances of the same class, and no pruning is required. More importantly, when splitting any node in the tree, it randomly selects a portion of the predictors, and the predictor with the best gini value is selected from this portion of predictors for splitting.

[0005] When scoring a data instance, Random Forest uses majority voting to combine the predictions from individual trees. For example, consider a target variable with 3 classes (c1, c2, and c3) and 1000 decision trees in the forest. Suppose the number of votes for the 3 classes is 200, 300, and 500, respectively. Then, Random Forest will report a vector of class probabilities for the instance being scored, i.e., [0.2, 0.3, 0.5]. It will also report the label prediction for c3 because it has the maximum predicted probability.

[0006] If the instance is a training instance, Random Forest provides the option of generating an Out-of-Bag (OOB) prediction. Such a prediction is the result of counting the votes of trees that have not yet included the instance in their bootstrap samples. Continuing with the above example, if there are 400 trees that have not yet used the instance for training, and the number of votes for c1, c2, and c3 are 100, 100, and 200, respectively, then the OOB prediction (probability vector) for the instance will be [0.25, 0.25, 0.50]. The label prediction will be c3 because it corresponds to the maximum probability. The OOB accuracy of the forest is calculated by comparing the instance label with the OOB label prediction for the training data.

[0007] The gcForest method uses cross-validation to generate new feature information. Although cross-validation is effective, it also brings challenges. First, the gcForest method includes multiple random forest learners N in each layer, and it requires k-fold cross-validation for each learner. Thus, each layer requires a total of N*k learners. When the data set is large, performance problems may arise because the number of learners increases with the size of the data set, and building even a single learner is expensive. Very powerful computing facilities will be required. Second, due to cross-validation and the need for a validation data set derived from the training data set, the training data may not be fully used. The gcForest method divides the data into training samples and validation samples, and stops training if the accuracy of the validation samples cannot be improved. The use of validation samples makes the small data problem even worse. Summary of the Invention

[0008] Data can be analyzed by systems, methods, and computer program products utilizing deep random forests, wherein the deep random forests have a reduced set of hyperparameters and reduced tuning requirements. In one aspect, a machine learning model is automatically constructed by a random forest layer. A decision tree of the random forest is grown from a data set. Out-of-bag (OOB) predictions and class label predictions are determined for the random forest. The OOB predictions for each instance of the data set are appended to the data set. The model is expanded by adding more forest layers. Each new forest grows from an additional version of the data set generated by the previous layer. A merger layer is added to the last data append layer to produce a model output.

[0009] In one aspect, computational resources can be reduced by using an approach in which a single user-provided hyperparameter is the number of trees in each forest. In this aspect, a machine learning model can be automatically constructed by receiving a training dataset and a specified number of decision trees for each random forest. The specified number of decision trees are then grown from the training dataset. An OOB prediction and a class label are determined for each instance in the training dataset. The OOB prediction is appended to the training data for each instance, and the next layer of the forest is then generated using the appended dataset. The OOB label prediction is used to determine the OOB accuracy of each layer. Additional layers are added until the OOB accuracy stops improving significantly with each additional layer. A combiner is added to combine the outputs of the last additional forest layer and provide the model output.

[0010] In one aspect, a forest of decision trees is grown from a training dataset. The number of trees for each forest and the dimensionality of the class vectors that will be used to grow the trees are specified. The model has multiple layers, with each layer providing a forest, thereby reducing the computational resources required for classification. Out-of-bag (OOB) predictions for the forest are determined. The OOB predictions are appended to the dataset as new features for each dataset instance. Appending OOB predictions provides a way to add new feature information from each layer to subsequent layers without having to use a model that causes feature information to be lost. The OOB accuracy is determined for the forest. Additional layers of a single forest are added by repeating the above steps and using the additional dataset from the previous layer for training until the OOB accuracy of the new layer no longer improves significantly. The use of OOB accuracy eliminates the need for cross-validation in the analysis, further reducing computational resources. The outputs of the last forest are combined, and the entire model is used to analyze data from outside the training dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of a system according to one embodiment of the present invention is provided.

[0012] Figure 2 A flow chart depicting a sequence of operations according to one embodiment of the present invention is provided.

[0013] Figure 3 A data flow according to one embodiment of the present invention is depicted.

[0014] Figure 4 A cloud computing environment according to one embodiment of the present invention is depicted.

[0015] Figure 5 An abstract model layer according to one embodiment of the present invention is depicted. DETAILED DESCRIPTION

[0016] Deep neural network (DNN) structures have been successfully applied to machine learning applications. DNNs may include many hyperparameters and therefore may be difficult to tune to the problem and dataset at hand. DNNs may also be difficult to apply to applications involving small datasets because DNNs may require large datasets to train the model to an acceptable level of accuracy. The complexity and opacity of the operation of DNN structures may make it difficult to determine the optimal structure in terms of number of layers and nodes per layer to achieve the desired or necessary accuracy when developing or specifying the DNN structure without excessive experimentation. What is needed is a machine learning model structure that can be easily specified and tuned without being associated with a DNN, even when applied to small datasets, and that can automatically complete model development when optimal accuracy has been achieved.

[0017] The disclosed systems, methods, and computer program products provide for machine learning model development and training that is applicable to small datasets due to the use of bootstrapped training datasets that are alternatively selected from a provided training dataset. Embodiments of the present invention describe only a single hyperparameter for model development - the number of random trees per forest. Embodiments of the present invention provide a simplified structure of a single random forest per layer and automatically stop model development when further improvement in accuracy is unlikely, thereby providing an accurate model with relatively low computational complexity. In one embodiment, models with more than one forest per layer can be generated. Such forests require additional computational resources but do not show a significant improvement in model accuracy.

[0018] In one embodiment, a computer-implemented method for developing and training a model for analyzing data may begin with the development and construction of a machine learning model. The model may be used for classification or regression analysis. Model development is based on a training data set associated with the machine learning task for which the model is to be used. The training data may be provided via a network, obtained from local sensors, or provided via a computer-readable medium.

[0019] Data can include numeric, textual, audio, video, or image data. Data can be associated with location, voice, music, entertainment, healthcare, financial information, vehicle data, logistics data, sales data, or other data types that require machine learning analysis.

[0020] The number of random decision trees per forest required by the model must be specified. This number can be specified by user input or can be set to a default value based on previous model development work. In one embodiment, this number can be specified as five hundred trees. In one embodiment, this number can be randomly selected using a random number generator or a pseudo-random number generator. As each layer / forest is added to the model, the number of trees per forest can remain constant. In one embodiment, the number of trees per forest can vary for each layer / forest.

[0021] Each decision tree is grown from bootstrapped data samples selected with replacement from the training dataset. The bootstrapped samples are comparable in size to the training dataset. This selection with replacement provides a way to grow trees from the training dataset while reducing the risk of overfitting the tree to the training data. For example, for the training dataset [1, 2, 3, 4, 5, 6], the bootstrapped samples selected with replacement could include [1, 2, 2, 3, 5, 6].

[0022] Each tree is fully grown from its corresponding bootstrap sample until each node of the tree contains only instances of the same class, and no pruning is required. As each tree grows, the random selection of class predictors defines the bifurcation of the tree nodes, where the node is defined by the subset with the lowest Gini impurity value. The dimensionality of the class vector can be specified as a parameter for building the model, or the model can be developed by considering a range of possible class vector dimensionality values. The Gini impurity value reflects the probability that a randomly selected data instance from the dataset is mislabeled. A Gini impurity value of zero indicates that the probability of the instance being mislabeled is 0, indicating that all instances at that node belong to the same class.

[0023] For example, the training data set includes one thousand data instances, each of which includes four instance attributes d, e, f, g and a classification label c. During the tree growth process, the system evaluates the results of using different combinations of instance attributes to separate the data at the root node to partition the data set. The evaluation selects the attribute combination with the lowest probability of mislabeling a random data instance, that is, the lowest class impurity. For this example, the root node is partitioned based on the value of attribute d. Each possible value of attribute d is defined as a branch of the root node. For example, if attribute d has four different values d1, d2, d3, d4, four branches are defined from the root node. The evaluation process is then performed on each node defined by the attribute d branch. Similarly, the attribute or attribute combination that produces the data partition that is least likely to mislabel a random data instance is selected to define the branch of each node. This selection may be different for each node defined by the original bifurcation of the data. For example, the combinations d1, e; d2, e; d3, f; and d4, g; define the next set of branches. This process continues on each branch that still contains labeled data for more than a single class, has more than the minimum number of instances on that branch, has not reached the specified maximum number of node levels, or whose class impurity cannot be improved by another fork.

[0024] In one embodiment, out-of-bag (OOB) predictions can be calculated for a random tree forest. The OOB predictions constitute a vector that includes dimensions derived from the sum of the votes of all random trees that do not have that particular instance as part of their bootstrapped dataset for a particular training dataset data instance. For example, consider a forest with five hundred random decision trees analyzing data with three classes, c1, c2, and c3. In this example, for two hundred trees, instance 1 of the training dataset is not part of the bootstrapped dataset. The evaluation of instance 1 by these two hundred trees produces one hundred classes, c1, fifty classes, c2, and fifty classes, c3. The OOB vector of the forest for instance 1 will be [0.5, 0.25, 0.25], and these three dimensions represent the probability of that class being selected by the tree. The OOB label prediction for each instance is also determined to be the highest probability class from the OOB prediction vector. For this example, the OOB label prediction will be c1.

[0025] In one embodiment, the OOB predicted class vector can be attached to the data instance as a new feature of the data instance. In this example, instance 1 of the dataset has the vector [0.5, 0.25, 0.25] attached to the data instance as a new feature determined by the forest. This attached dataset is then passed to the next layer of the model to grow the decision tree of the single forest at that layer.

[0026] In one embodiment, the OOB accuracy of the forest / layer is also calculated. OOB accuracy is calculated as the comparison of the OOB label predictions to the actual data labels of the labeled data. In this example, instance 1, labeled c1 and with an OOB prediction of [0.5, 0.25, 0.25], has an OOB label prediction of c1, which is correctly labeled. The OOB accuracy of the entire forest for the entire dataset is calculated as the proportion of instances correctly classified by the label predictions.

[0027] In one embodiment, a second forest / layer is generated using an additional dataset output from the original layer as a new training dataset. Each of the predetermined number of random trees in the second forest is grown from a bootstrapped dataset, alternately taken from the additional training dataset. An OOB prediction is calculated for the new forest and further appended to the training dataset. The OOB accuracy of the new forest is determined and compared to the OOB accuracy of the previous forest / layer of the model.

[0028] Continue iterating: forest generation, data appending, OOB prediction, and OOB accuracy calculation until the current layer OOB accuracy no longer changes significantly from the previous layer. In one embodiment, a change in OOB accuracy greater than 0.005% is considered a significant improvement.

[0029] In one embodiment, the output of each layer of the forest is appended to the dataset, so that each instance of the dataset has a new feature added to each layer of the model used. In this embodiment, by adding new features to each data instance, each subsequent layer is provided with new information about each instance, rather than compressing the new information determined about the instance into the supplementary data model of the subsequent layer. Due to model fidelity issues, such supplementary models may result in a loss of feature information because the model may smooth feature information across data instances instead of appending specific new feature information to each instance.

[0030] When adding a new forest does not significantly improve the OOB accuracy, a combiner function is added to combine the outputs of the last layer. Another random forest can be used as a combiner without computing OOB predictions or OOB accuracy. In one embodiment, a gradient boosting function, such as XGBoost, can be used as a function to combine the outputs of the last forest. The addition of the combiner takes advantage of the additional features added to the dataset by the last forest added to the model. A model without the combiner would produce results that do not take into account this final OOB predictor added to the dataset and provide lower prediction accuracy. After the combiner is added to the layer / forest, the model can be used for the desired purpose in a machine learning context to analyze data outside the training dataset.

[0031] Example:

[0032] In one embodiment, a machine learning model is constructed from a dataset. A random forest of decision trees is grown from the dataset. The out-of-band predictions for the forest are determined and appended to the dataset. Additional forests of decision trees are grown from the appended dataset. A combiner is added to the model to combine the outputs of the additional forests and provide the model output.

[0033] In one embodiment, a training data set is received along with a specification of the number of decision trees for each forest. A first random forest having a specified number of decision trees is grown from the data set. OOB predictions and class labels are determined using the random forest. The OOB predictions are appended to the training data set. The OOB accuracy of the forest is determined. An additional forest having a specified number of decision trees is grown from the appended data set. OOB predictions and class labels are determined for the additional forest. The OOB predictions are appended to the already appended data set. The OOB accuracy of the additional forest is determined and compared to the OOB accuracy of the original forest. This process of growing forests, determining OOB predictions, class labels, and accuracy, appending the predictions to the data set, and comparing the accuracy to the accuracy of the previous forest is continued until the OOB accuracy is not significantly improved (<0.005%) by the new forest. The merged structure is added to the final forest to provide the output of the model.

[0034] In one embodiment, a training dataset is received along with a specification of the number of decision trees for each forest and a specification of the class vectors for the decision trees. A model is then constructed having multiple sequential layers. Each layer includes a single random forest to reduce the computational resource requirements when constructing the model. A random forest having a specified number of decision trees is grown from the dataset. OOB predictions and class labels are determined using the random forest. The OOB predictions are appended to the training dataset. The OOB accuracy of the forest is determined. An additional forest having a specified number of decision trees is grown from the appended dataset. OOB predictions and class labels are determined for the additional forest. The OOB predictions are appended to the already appended dataset. The OOB accuracy of the additional forest is determined and compared to the OOB accuracy of the original forest. This process of growing forests, determining OOB predictions, class labels, and accuracy, appending the predictions to the dataset, and comparing the accuracy to the accuracy of the previous forest is continued until the OOB accuracy is not significantly improved (<0.005%) by the new forest. The merged structure is added to the last forest to provide the output of the model.

[0035] Experimental results:

[0036] An embodiment of the present invention was constructed and compared to a standard using a standard dataset: random forest, XGBoost, and gcforest models. For this experiment, the number of trees in each forest was specified to be five hundred, the number of randomly selected attributes was √d (d is the number of attributes of the data instance), and each tree was grown to a pure leaf node. An embodiment of the present invention was constructed using each of the random forest and the XGBoost merger for XGBoost, using default settings. For GCForest, each layer included four fully random forests and four regular random forests. A three-fold class vector was used for class vector generation.

[0037] Each data set was divided into a training sample (50%) and a test sample (50%). The training sample was used to train the model and the test sample was used to evaluate the trained model. For each data set, each model type was built, trained and evaluated five times. The results are shown in Table 1 below. The embodiments of the present invention are labeled as Incremental Deep Forest with Random Forest Merger (IDF RF) and IDF with XGBoost Merger (IDF XGBoost). As shown in the table, for most of the standard data sets evaluated, the accuracy of the embodiments of the present invention exceeds the accuracy of the known machine learning model I.

[0038] Table 1: Model accuracy (%)

[0039]

[0040]

[0041] Figure 1 A schematic diagram of exemplary network resources associated with practicing the disclosed invention is provided. The present invention may be practiced in a processor of any disclosed element that processes an instruction stream. As shown, a networked client device 110 is wirelessly connected to a server subsystem 102. A client device 104 is wirelessly connected to the server subsystem 102 via a network 114. The client devices 104 and 110 include a machine learning program (not shown) and sufficient computing resources (processor, memory, network communication hardware) to execute the program. Figure 1 As shown, server subsystem 102 includes server computer 150 . Figure 1 A block diagram of components of a server computer 150 within a networked computer system 1000 is depicted according to an embodiment of the present invention. It should be understood that Figure 1 This merely provides an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented.Many modifications to the depicted environments are possible.

[0042] The server computer 150 may include a processor 154, a cache 162, a memory 158, a persistent storage 170, a communication unit 152, an input / output (I / O) interface 156, and a communication fabric 140. The communication fabric 140 provides communication between the cache 162, the memory 158, the persistent storage 170, the communication unit 152, and the input / output (I / O) interface 156. The communication fabric 140 may be implemented using any architecture designed to transfer data and / or control information between a processor (e.g., a microprocessor, a communication and network processor, etc.), system memory, peripheral devices, and any other hardware components within the system. For example, the communication fabric 140 may be implemented using one or more buses.

[0043] Memory 158 and persistent storage 170 are computer-readable storage media. In this embodiment, memory 158 includes random access memory 160 (RAM). In general, memory 158 may include any suitable volatile or non-volatile computer-readable storage media. Cache 162 is a fast memory that enhances the performance of processor 154 by retaining recently accessed data and data near recently accessed data from memory 158.

[0044] Program instructions and data for practicing embodiments of the present invention, such as machine learning program 175, are stored in persistent storage 170 for execution and / or access by one or more of the corresponding processors 154 of server computer 150 through cache 162. In this embodiment, persistent storage 170 comprises a magnetic hard drive. Alternatively, or in addition to a magnetic hard drive, persistent storage 170 may comprise a solid-state hard drive, a semiconductor memory device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0045] The media used by persistent storage 170 may also be removable. For example, a removable hard drive may be used for persistent storage 170. Other examples include optical and magnetic disks, thumb drives, and smart cards, which are inserted into a drive for transfer to another computer-readable storage medium that is also part of persistent storage 170.

[0046] In these examples, communications unit 152 provides communications with other data processing systems or devices, including resources of client computing devices 104 and 110. In these examples, communications unit 152 includes one or more network interface cards. Communications unit 152 may provide communications using one or both of physical and wireless communication links. Software distribution programs and other programs and data used to implement the present invention may be downloaded to persistent storage 170 of server computer 150 via communications unit 152.

[0047] The I / O interface 156 allows for input and output of data to and from other devices that may be connected to the server computer 150. For example, the I / O interface 156 may provide a connection to an external device 190, such as a keyboard, a keypad, a touch screen, a microphone, a digital camera, and / or some other suitable input device. The external device 190 may also include portable computer-readable storage media, such as a thumb drive, a portable optical or magnetic disk, and a memory card. Software and data used to practice embodiments of the present invention, such as the machine learning program 175 on the server computer 150, may be stored on such a portable computer-readable storage medium and may be loaded onto the persistent storage 170 via the I / O interface 156. The I / O interface 156 is also connected to the display 180.

[0048] Display 180 provides a mechanism for displaying data to a user and may be, for example, a computer monitor. Display 180 may also be used as a touch screen, such as the display of a tablet computer.

[0049] Figure 2A flowchart 200 is provided showing exemplary activities associated with an embodiment of the present invention. After the program is started, a training dataset is received at 210. The training dataset can include any form of data associated with the machine learning task for which the desired model is to be used. Examples include numeric, character, audio, video, and image data, and combinations thereof. The number of random trees for each forest is received at 220. This number can be selected by the user, automatically determined based on the nature of the dataset and the machine learning task, or automatically selected using a random or pseudo-random number generator. The random trees are grown at 230. Each tree can be grown from a bootstrap data sample of comparable size to the training dataset, selected from the training dataset with replacement. The bootstrap data can be selected without replacement and using a smaller set than the training data, but such selections can reduce the accuracy of the resulting model and lead to overfitting of the model to the training dataset. An out-of-bag (OOB) prediction and label prediction are determined for each instance of the training data at 240. The OOB prediction provides new feature information about each data instance and is appended to the corresponding instance in the dataset at 250. The OOB accuracy is determined for the complete forest at 260. OOB label predictions are taken for each instance and the correct predictions are aggregated across the entire forest and the entire training dataset to calculate the OOB accuracy. The OOB accuracy is compared to the OOB accuracy from the previous layer of the model at 270. If there is a significant improvement in OOB accuracy (in one embodiment, an improvement of >0.005% constitutes a significant improvement), the method returns to step 230, another layer / forest is grown, and added to the model. In one embodiment, after a significant improvement, the method returns to step 220 and the number of trees in the new layer / forest can be determined, which can be different from the previous layer / forest. If there is no significant improvement in OOB accuracy, a merger is added to the model at 280 to aggregate the output of the last layer / forest for use. The model of this method only includes the number of forests / layers necessary to optimize OOB accuracy. The model passes all new feature information (OOB predictions) to each subsequent layer so that no new feature information determined by the model is lost in this pass.

[0050] Figure 3An example of the evolution of a training dataset as a model is constructed is provided. As shown, a dataset 300 including data instance X is provided as input to forest 1 310. For each data instance in dataset 300, a class vector P1 is determined by forest 1 310 for instance X and appended to instance X, generating an additional dataset 320. The additional dataset 320 is then used as input to forest 2 330, which generates a class vector P2 for instance X. The class vector P2 is appended to the instance, generating an additional dataset 340. This process continues until the OOB accuracy of the new forest is not significantly improved compared to the previous forest. A merger 350 is added to the model to aggregate the output of the last added forest, represented in the figure by the additional dataset 350, which contains all the additional class vectors, including the last class vector Pn. The output 370 of the merger 360 constitutes the class prediction for the data instance.

[0051] It should be understood that although the present disclosure includes detailed descriptions about cloud computing, the implementation of the teachings cited herein is not limited to cloud computing environments. Instead, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0052] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0053] Features are as follows:

[0054] On-demand self-service: Cloud consumers can automatically and unilaterally provision computing capabilities, such as server time and network storage, on demand without requiring manual interaction with the service provider.

[0055] Broad Network Access: Functionality is provided over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, and PDAs).

[0056] Resource pooling: Providers' computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated as needed. Location independence is important because consumers typically have no control or knowledge of the exact location of the provided resources, but may be able to specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0057] Rapid elasticity: In some cases, capabilities can be provisioned quickly and elastically to scale up and down quickly and released quickly to scale back in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased at any time and in any quantity.

[0058] Measured services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0059] The service model is as follows:

[0060] Software as a Service (SaaS): The ability provided to consumers is to use the provider's applications running on a cloud infrastructure. These applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functionality, with the possible exception of limited user-specific application configuration settings.

[0061] Platform as a Service (PaaS): The capability provided to consumers is to deploy consumer-created or acquired applications on cloud infrastructure, using programming languages and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but do have control over the deployed applications and possibly the configuration of the application hosting environment.

[0062] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, networking, and other basic computing resources on which they can deploy and run arbitrary software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0063] The deployment model is as follows:

[0064] Private cloud: Cloud infrastructure is operated solely for an organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0065] Community cloud: The cloud infrastructure is shared by multiple organizations and supports a specific community with common concerns (e.g., mission, security requirements, policy, and compliance considerations). It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0066] Public cloud: Cloud infrastructure is available to the general public or large industry groups and is owned by the organization that sells cloud services.

[0067] Hybrid cloud: A cloud infrastructure composed of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0068] The cloud computing environment is service-oriented and focuses on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is the infrastructure consisting of a network of interconnected nodes.

[0069] Now see Figure 4 , depicting an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10, with which local computing devices used by cloud consumers (such as personal digital assistants (PDAs) or cell phones 54A, desktop computers 54B, laptop computers 54C, and / or automobile computer systems 54N) can communicate. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as in the private clouds, community clouds, public clouds, or hybrid clouds described above, or a combination thereof. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as a service for which cloud consumers do not need to maintain resources on their local computing devices. It should be understood that Figure 4 The types of computing devices 54A-N shown in are intended to be illustrative only, and computing node 10 and cloud computing environment 50 may communicate with any type of computerized device over any type of network and / or network-addressable connection (e.g., using a web browser).

[0070] Now see Figure 5 , showing the cloud computing environment 50 ( Figure 4 ) provides a set of functional abstraction layers. It should be understood in advance that Figure 5 The components, layers, and functions shown in the figure are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0071] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: host 61; server 62 based on RISC (Reduced Instruction Set Computer) architecture; server 63; blade server 64; storage 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0072] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71 ; virtual storage 72 ; virtual networks 73 , including virtual private networks; virtual applications and operating systems 74 ; and virtual clients 75 .

[0073] In one example, the management layer 80 may provide the functionality described below. Resource provisioning 81 provides dynamic acquisition of computing resources and other resources for performing tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking when resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection of data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management so that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-arrangement and procurement of cloud computing resources for future requirements anticipated according to the SLA.

[0074] The workload layer 90 provides examples of functionality that can take advantage of a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics processing 94; transaction processing 95; and machine learning programs 175.

[0075] The present invention may be a system, method, and / or computer program product at any possible level of technical detail integration. The present invention may be advantageously practiced in any system (single or parallel) that processes an instruction stream. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0076] A computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punched card or a raised structure in a groove with instructions recorded thereon), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by a fiber optic cable), or an electrical signal transmitted by a wire.

[0077] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards these computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0078] The computer-readable program instructions for performing the operation of the present technical solution can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, the configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and procedural programming languages, such as "C" programming language or similar programming languages. The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or executed entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In certain embodiments, electronic circuits (including for example programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs)) can execute computer-readable program instructions to personalize the electronic circuits by using the state information of computer-readable program instructions, so as to perform the various aspects of the present technical solution.

[0079] Various aspects of the present technical solution are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the technical solution. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0080] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine that, when executed by the processor of the computer or other programmable data processing apparatus, creates a means for implementing the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing apparatus, and / or other device to function in a specific manner, such that the computer-readable storage medium having the instructions stored therein comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams.

[0081] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other apparatus that causes a series of operating steps to be performed on a computer, other programmable apparatus or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus or other device implement the functions and actions specified in one or more blocks in the flowchart and / or block diagram.

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the systems, methods and computer program products according to the different embodiments of the present technical solution. To this end, each box in the flowchart or block diagram may represent a part of a module, segment or instruction, which includes one or more executable instructions for implementing the specified logical function. In some alternative embodiments, the functions marked in the box may not occur in the order marked in the figure. For example, depending on the functions involved, the two boxes shown in succession can actually be executed substantially simultaneously, or these boxes can sometimes be executed in the opposite order. It will also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented by a system based on dedicated hardware, which performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0083] The description of the various embodiments herein has been presented for purposes of illustration and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the marketplace, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for developing and training a model for analyzing data, the method comprising: Construct the model by following these steps: growing, by one or more computer processors, a random forest of decision trees from a dataset, wherein the dataset comprises dataset instances of image and / or speech data, each random forest being grown based on the image and / or speech data; determining, by one or more computer processors, out-of-bag (OOB) predictions for instances of the dataset of the random forest; appending, by one or more computer processors, the out-of-bag (OOB) prediction to the dataset as a probability vector as a new feature for instances of the dataset; growing, by one or more computer processors, additional random forests using the data set with the additional OOB predictions; and The outputs of the additional random forests are combined with a combiner by one or more computer processors to serve as class predictions for dataset instances in the dataset.

2. The method according to claim 1, wherein Growing the random forest of decision trees using a dataset includes growing the decision trees using bootstrap samples to grow each tree, the bootstrap samples being taken with replacement from the dataset.

3. The method according to claim 1, further comprising: The OOB accuracy of each random forest was determined, and random forests were added until the OOB accuracy did not improve significantly.

4. The method according to claim 1, wherein The combiner includes a structure selected from the group consisting of a random forest and a gradient boosting structure.

5. The method according to claim 1, wherein The model comprises a sequence of layers, each layer comprising a single random forest.

6. The method according to claim 1, comprising: receiving, by one or more computer processors, a training data set; Determining the OOB accuracy of the random forest; determining, by one or more computer processors, an OOB accuracy of the additional random forest; The OOB accuracy of the random forest and the additional random forest are compared, by one or more computer processors.

7. The method according to claim 6, comprising: The model is constructed with sequential layers, each consisting of a single random forest, by following these steps: receiving, by one or more computer processors, a training data set; receiving, by one or more computer processors, a determined number of trees and a class vector specification for each forest; growing, by one or more computer processors, the determined number of trees for the first forest using the training data set; determining, by one or more computer processors, a first out-of-bag (OOB) prediction for the first forest; appending, by one or more computer processors, the OOB prediction to the dataset; determining, by one or more computer processors, an OOB accuracy of the first forest; increasing, by one or more computer processors, the determined number of trees for an additional forest using the training data set with the additional OOB predictions; determining, by one or more computer processors, additional OOB predictions for the additional forests; appending, by one or more computer processors, additional OOB predictions to the data set; determining, by one or more computer processors, an additional OOB accuracy for the additional forest; adding, by one or more computer processors, forests until the additional OOB accuracy does not improve; and The outputs of the additional forests are combined by one or more computer processors.

8. A computer program product for developing and training a model for analyzing data, the computer program product comprising program instructions which, when executed by one or more computer processors, cause the processors to perform the steps of the method according to any one of claims 1 to 7.

9. A computer system for developing and training a model for analyzing data, the computer system comprising: one or more computer processors; one or more computer-readable storage devices; Program instructions stored on the one or more computer-readable storage devices for execution by the one or more computer processors, the program instructions, when executed by the one or more computer processors, causing the computer processors to perform the steps of the method according to any one of claims 1 to 7.

Images