A data-driven approach to performance-based project management
A data-driven approach using machine learning techniques automates project analysis and prediction, addressing manual inefficiencies in conventional methods to improve project management by uncovering hidden attributes and optimizing resource allocation.
Patent Information
- Application Number
- JP2023551207
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-17
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-03-17
AI Technical Summary
Conventional project management systems rely heavily on manual analysis and estimation, which are time-consuming, subjective, and prone to risks, failing to systematically analyze all project attributes, human resources, and constraints, leading to suboptimal performance and performance prediction inaccuracies.
Implementing a data-driven approach using unsupervised and supervised machine learning techniques for project profiling, performance monitoring, and optimization, including feature extraction, self-profiling algorithms, and collaborative filtering to automate project analysis, monitoring, and prediction.
Enhances project management by discovering hidden attributes, improving performance prediction accuracy, and optimizing resource allocation, thereby reducing subjectivity and time consumption, and enhancing overall project performance.
Smart Images

Figure 0007590593000001 
Figure 0007590593000002 
Figure 0007590593000003
Abstract
Description
[Technical field]
[0001] The present disclosure is directed to project management systems, and more specifically, to the use of data-driven approaches to facilitate performance-based project management. [Background technology]
[0002] A project (or program) is any undertaking carried out individually or collaboratively through carefully planned study or design (e.g., by a project team) to achieve a specific goal.
[0003] Projects can have various attributes. For example, projects can be internal (e.g., product development) or external (e.g., customer solutions). Projects can also be in various categories such as software, services, and analytics. Projects can be in various industries such as manufacturing, transportation, and healthcare, among others. Projects vary in duration, ranging from weeks, months, to years. Projects can include one-time tasks (e.g., consulting) or ongoing work (e.g., phases I, II, III of consulting; versions 1, 2, 3 of product development).
[0004] Given a project, a manager or stakeholder needs to understand and analyze the above-listed project attributes and plan the management and execution of the project accordingly. The manager or stakeholder can analyze each attribute separately, combine some attributes manually, and gain insights based on his or her own experience and expertise.
[0005] To be effectively optimized, project management includes several key elements to consider: A project can be managed as a series of events, which may be a set of interrelated tasks performed over a fixed period of time and within a certain cost range along with other constraints and risks. Before a project officially begins, managers or stakeholders may need to estimate performance, such as the profit margins that will be achieved. Managers or stakeholders may need to plan the management and execution of the project to achieve the best results and performance. After a project begins, managers may need to monitor the project's progress and adjust the project execution plan as needed. Summary of the Invention [Means for solving the problem]
[0006] Some limitations and restrictions of conventional systems and methods are discussed below. The implementations described herein introduce techniques to address the following problems:
[0007] In the first problem of the related art, a project is manually considered and analyzed based on project attributes. The understanding and analysis results are used to obtain project insights. Some important hidden elements of the project may not be discovered. Such a consideration and analysis process may be time-consuming, subjective, and prone to risk. In order to discover hidden elements for business insights in projects, it is necessary to automate the analysis and profiling of projects in a data-driven manner.
[0008] In the second problem of the related art, projects are manually monitored by periodic inspections or milestones. Project performance is manually analyzed and evaluated, and some adjustments may be made to the execution plan. Such monitoring and evaluation is time-consuming, subjective, and prone to risk. There is a need to have an automated and standard process or methodology for monitoring and evaluating project progress in a data-driven manner.
[0009] In the third problem of the related art, the performance of a project is estimated manually based on the understanding and analysis of some attributes of the project, some aspects of human resources, and some constraints. However, not all attributes of the project, human resources, and constraints are systematically analyzed and summarized. Not all available information is appropriately utilized or used. Furthermore, the estimation is usually related to the obtained final performance, such as the obtained profit rate, based on the information available before the start of the project. Such manual estimation based on experience and expertise is subjective, time-consuming, and prone to risk. Therefore, there is a need to have an automated data-driven method that can estimate and predict the performance of a project by incorporating all available information. The data-driven method can also predict the performance of the project at each milestone by using all available information (including project execution information) prior to the milestone. There is also a need to provide an explanation for the prediction.
[0010] In the fourth problem of the related art, project managers design plans to execute projects by scheduling to complete the project by a scheduled date, focusing mostly on resource availability rather than on performance within the project. All attributes of the project, all aspects of human resources, and constraints are not systematically analyzed and summarized. As a result, the project may not achieve the best performance. Such manual planning based on experience and expertise is subjective, time-consuming, and prone to risk. Therefore, there is a need for a data-driven approach to systematically analyze and summarize all relevant factors and optimize project performance.
[0011] To address the above four problems, the implementation example described herein includes several aspects. One aspect includes project self-profiling, in which several unsupervised learning schemes are introduced to automatically learn and derive hidden attributes of a project. Such techniques help automate project analysis and profiling through data-driven techniques and discover hidden factors related to business insights and downstream tasks. Another aspect includes performance monitoring, in which data-driven techniques are introduced to capture project performance transitions with respect to multiple key performance indicators (KPIs) among milestones. The implementation example also introduces techniques for predicting multiple performance KPIs at each milestone. Another aspect includes project performance prediction and root cause analysis, in which machine learning techniques are introduced to predict project performance at each milestone and explain the predictions. Another aspect includes project performance optimization, in which data-driven techniques are introduced to optimize project performance by identifying the best project team for the project.
[0012] Aspects of the disclosure may include a computer-implemented method for facilitating project management that may include, for a project input including project data and employee data, performing feature extraction on the project data and employee data to generate features, running a self-profiling algorithm consisting of unsupervised machine learning on the generated features to derive clusters and anomalies for the project, performing a performance monitoring process on the generated features to determine probabilities of key performance indicator values at milestones, and running a supervised machine learning model on the generated features, derived clusters, derived anomalies, and probabilities of key performance indicator values at milestones to generate a predicted performance value for the project. The computer-implemented method may be executed as a computer program on one or more processors and stored on a non-transitory readable medium.
[0013] Aspects of the disclosure may include a system for facilitating project management that may include, for a project input including project data and employee data, means for performing feature extraction on the project data and employee data to generate features, means for running a self-profiling algorithm consisting of unsupervised machine learning on the generated features to derive clusters and anomalies for the project, means for running a performance monitoring process on the generated features to determine probabilities of key performance indicator values at milestones, and means for running a supervised machine learning model on the generated features, the derived clusters, the derived anomalies, and the probabilities of key performance indicator values at milestones to generate a predicted performance value for the project.
[0014] Aspects of the disclosure further include a computer-implemented method that may include: generating a matrix from project performance data, the matrix being comprised of rows of employee-related data and columns of project-related data across various dimensions and including values indicative of performance scores derived from the project performance data; for missing values in the matrix, generating missing values in the matrix from a training process; for a new project input, identifying rows and columns that match employee-related data in the new project and project-related data for the new project, aggregating values corresponding to the identified rows and columns, ranking employee groups from the employee-related data according to the aggregated values, and selecting an employee group from the employee-related data with the highest rank for execution of the new project. The computer-implemented method may be executed as a computer program on one or more processors and stored on a non-transitory readable medium.
[0015] Aspects of the present disclosure further include a system that may include a means for generating a matrix from the project performance data, the matrix being composed of rows of employee-related data and columns of project-related data across various dimensions and including values indicative of a performance score derived from the project performance data; for missing values in the matrix, a means for generating missing values in the matrix from a training process; for an input of a new project, a means for identifying rows and columns that match employee-related data in the new project and project-related data for the new project; a means for aggregating values corresponding to the identified rows and columns; a means for ranking employee groups from the employee-related data according to the aggregated values; and a means for selecting an employee group from the employee-related data having the highest rank with respect to execution of the new project.
[0016] Aspects of the disclosure further include a computer-implemented method for executing a self-profiling algorithm comprised of unsupervised machine learning on the generated features to derive clusters and anomalies of Internet of Things (IoT) data, the method including: executing each unsupervised machine learning model algorithm from a set of unsupervised learning model algorithms on the generated features, determining for each unsupervised machine learning model algorithm a best of the unsupervised machine learning models having an associated best parameter set, and determining a best unsupervised model across the set of unsupervised machine learning model algorithms from the best of the unsupervised machine learning models for each unsupervised machine learning model algorithm. The computer-implemented method may be executed as a computer program on one or more processors and stored on a non-transitory readable medium.
[0017] Aspects of the disclosure further include a system for running a self-profiling algorithm comprised of unsupervised machine learning on the generated features to derive clusters and anomalies in Internet of Things (IoT) data, the system including: means for running each unsupervised machine learning model algorithm from a set of unsupervised learning model algorithms on the generated features; means for determining a best of the unsupervised machine learning models having an associated best parameter set for each unsupervised machine learning model algorithm; and means for determining a best unsupervised model across the set of unsupervised machine learning model algorithms from the best of the unsupervised machine learning models for each unsupervised machine learning model algorithm.
[0018] Aspects of the present disclosure further include a computer-implemented method for executing a self-profiling algorithm comprised of unsupervised machine learning on the generated features to derive clusters and anomalies of Internet of Things (IoT) insurance data, the method including: a) applying a set of unsupervised machine learning model algorithms to the generated features to generate unsupervised machine learning models for each of the unsupervised machine learning algorithms, b) appending unsupervised output from the unsupervised machine learning models to the features, and c) repeating steps a) and b) until a termination criterion is met. The computer-implemented method may be executed as a computer program on one or more processors and stored on a non-transitory readable medium. [Brief description of the drawings]
[0019] [Figure 1] FIG. 1 illustrates a solution architecture for performance-based project management according to one implementation.
[0020] [Diagram 2] FIG. 2 illustrates framework components for facilitating unsupervised machine learning with hyperparameter tuning according to one example implementation.
[0021] [Diagram 3] FIG. 3 illustrates an example of an iterative learning scheme according to one implementation.
[0022] [Figure 4] FIG. 4 illustrates an example of a recursive learning scheme according to one implementation.
[0023] [Diagram 5] FIG. 5 illustrates a milestone-based KPI transition network according to one implementation example.
[0024] [Figure 6] FIG. 6 shows a tabular construction of a multi-task KPI prediction problem according to one implementation example.
[0025] [Figure 7] FIG. 7 illustrates an example of a project performance forecast by milestone according to one implementation.
[0026] [Figure 8] FIG. 8 illustrates an example of a multi-level, multi-dimensional collaborative filtering matrix for optimizing a project team according to one implementation.
[0027] [Figure 9] FIG. 9 illustrates an example computing environment having an example computer device suitable for use with some implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0028] The following detailed description shows the drawings and implementation details of the present application. Redundant element reference numbers and descriptions between the drawings are omitted for clarity. The terms used throughout the description are shown by way of example and are not intended to be limiting. For example, the use of the term "automatic" may include a fully automatic implementation or a semi-automatic implementation including user or administrator control over certain aspects of the implementation, depending on the desired implementation of the person skilled in the art practicing the implementation of the present application. The selection may be made by a user through a user interface or other input means, or may be implemented by a desired algorithm. The implementations described herein may be utilized alone or in combination, and the functionality of the implementations may be implemented by any means depending on the desired implementation.
[0029] 1 illustrates a solution architecture for performance-based project management according to one implementation. The solution architecture may include various components as described below and throughout this disclosure. Inputs 101 are provided via components 102-106 to generate predicted performance and optimal teams for a project.
[0030] Data preparation 110 collects and prepares data for subsequent insight discovery 120 and performance modeling 130 components, which may include the following components: Input 101 from multiple sources is collected and used as input in the solution: In one example, project input 101 may include, but is not limited to, project data such as project attributes and performance (final performance and milestone based performance) and project opportunity management, employee data such as employee attributes and performance (human resources and project based), etc. Feature engineering component 102 is used to derive features that are used to build models in the insight discovery and performance modeling components.
[0031] Insight discovery 120 may include the following components to discover insights independent of features or project performance: Self-profiling component 103 discovers insights about projects by unsupervised learning techniques on project and employee features (e.g., non-performance data), Performance monitoring component 104 discovers insights about projects by milestone-based performance transition networks or multi-task supervised machine learning models on KPI-based performance data.
[0032] Performance modeling 130 includes the following components for optimizing project performance in two scenarios: Given project data and project team, performance prediction 105 predicts the milestone-based performance of the project. The output of this component is the predicted performance and optimal team for the project. Given project data and no project team, performance optimization 106 finds the project team that can achieve the best project performance. The output of this component is the project team with optimal project performance.
[0033] The following discussion provides a detailed discussion of each component in the solution architecture.
[0034] The following discussion is directed to how to collect and prepare data for the data-driven approach. Data Preparation 110
[0035] Data 101 from multiple sources is collected. Depending on the availability of data, the following data may be collected and used to build the solution: project data, employee data, and client data.
[0036] Project data specifies information for obtaining, defining, describing, and monitoring projects. Project data includes some or all of the following components, but is not limited to: project description, various project attributes, project performance, project execution information, statement of work, and project opportunity management data. Project description is a description of the project. Project attributes include various attributes of each project. For example, projects can be internal (product development) or external (customer solutions), and can be in various categories (e.g., software, services, analytics, etc.), of various durations (e.g., weeks, months, or years) depending on the desired implementation, one-time tasks (consulting) or ongoing work (phases I, II, III of consulting, product development), proof of concept (PoC) projects, or formal consulting projects, etc. Project performance includes using various metrics or KPIs to measure project performance. KPIs can include measurable values that demonstrate how effectively the project team is achieving key business objectives for the project. Typically, a project can be divided into several milestones, and such KPIs can be measured at each milestone. Overall, you can also measure KPIs over the entire duration of the project.
[0037] KPIs may include, but are not limited to: Revenue KPIs measure all revenues and cash receipts of the project team derived from owning or operating the project. Expense KPIs measure the total funds required to complete the project or job which may include direct and indirect costs. Direct and indirect costs as well as more specific costs may be used as sub-KPIs as follows: Direct Costs (DC) are expenditures that are directly related to a specific task or project and may be identified to a specific cost center. Direct costs of a project include directly related efforts or expenditures such as employee, equipment and materials costs, and project management / engineering / procurement related costs such as outsourced costs, transportation and customs clearance, construction management / labor / equipment and supplies / subcontractor costs, etc.
[0038] Overhead costs are any costs that are not directly part of a specific task or project and cannot be precisely attributed to a specific cost center. Overhead costs may be fixed or variable. Overhead costs include administrative costs, labor costs, and security costs. These are costs that are not directly related to production.
[0039] Gross Margin Margin (GPM) is a KPI that measures the profitability of a project. GPM is the percentage of revenue that is real profit before adjusting for operating expenses such as marketing, general expenses, and salaries. An example of a calculation could be Gross Margin Margin = (Revenue - Expenses) / Revenue, where Expenses represents the Cost of Goods Sold (COGS).
[0040] Resource Utilization is a KPI that measures the difference between actual resource utilization and forecasted resource utilization.
[0041] Task Completion Rate is a KPI that measures the percentage of tasks completed out of the total work expected to be completed. Tasks may be measured within various workstreams including product management, development, quality control, etc. Such tasks may be used as sub-KPIs.
[0042] Client Satisfaction is a KPI that measures the level of satisfaction from customers.
[0043] Other KPIs may be utilized depending on the desired implementation and the disclosure is not limited thereto.
[0044] Other project data may include a Statement of Work (SOW), which is a document that captures and defines all aspects of the project. The SOW is a detailed document because it lays the groundwork for the project plan. The SOW may include an introduction to the project, the project objectives, the scope of work, a list of tasks, milestones, deliverables, schedule, etc.
[0045] Additionally, project opportunity management data includes information regarding obtaining a project, including the source of the opportunity, a timeline of opportunity stages, evaluation information (financial, technical, data, and risk), information regarding the sales team, payment terms, etc.
[0046] In addition, project data may include project execution information. As a project begins, several pieces of data related to project performance may be collected, including, but not limited to, data availability, data quality, development environment setup and availability, technology readiness, employee absenteeism, communication availability and quality, the KPIs just discussed at each milestone, etc.
[0047] With regard to employee data, a project team is formed of a group of employees. Each employee data defines information about each employee for workforce management. Employee data includes some or all of the following information, but is not limited to: Human Resources (HR) information includes all data about employees in an HR database, including employee demographic data, technical skills, interests, seniority, role, title, etc. Project specific data includes employee performance based on performance on completed projects or performance at each milestone of completed or ongoing projects.
[0048] Client data may include, but is not limited to, the following types of information: Clients may be internal (product development) or external (consulting clients). Clients may be from different industries (e.g., manufacturing, transportation, healthcare, etc.). Clients may be of different sizes (e.g., large, medium, or small) in terms of number of employees, revenue, or profits. Clients may be in different regions or countries. Clients may have different credits in terms of ease of payment. Other client data may also be incorporated depending on the desired implementation.
[0049] Once data is collected, features can be derived from the data via the feature engineering component 102. Features can be derived manually based on expert knowledge and data analysis. The implementations described herein focus on automatic feature engineering techniques. The following techniques belong to unsupervised learning, which does not use labels.
[0050] Principal Component Analysis (PCA) is a dimensionality reduction process by projecting each data point onto only the first few principal components to obtain a low-dimensional data set while preserving as much data variation as possible. PCA also has the effect of reducing noise caused by uncaptured variance. PCA makes downstream models more generalizable.
[0051] Zero-phase component analysis (ZCA) is another technique used for dimensionality reduction. ZCA performs a different rotation than PCA does. PCA is the best analysis if the goal is to compress the original data (because the principal components are sorted according to their explained variance). ZCA is the best analysis if the goal is to keep the transformed random vectors as similar as possible to the original.
[0052] Autoencoders are another technique for dimensionality reduction and noise reduction. Autoencoders use a type of artificial neural network to learn efficient data coding in an unsupervised manner. Autoencoders use data as both the input and output layers of the neural network as well as one or more hidden layers of smaller size. Autoencoders can be separated into two components: an encoder where the data is first encoded into a smaller size coding, and a decoder where the coding is decoded back to the original data. The effectiveness of an autoencoder is measured by how accurately the coding represents the original data, which is in turn measured by the accuracy of the neural network. Autoencoders come in different varieties: regularized autoencoders (sparse, denoising, and shrinking autoencoders), which are efficient and effective in learning representations for downstream tasks, and variational autoencoders, which are useful for generative models.
[0053] The PCA, ZCA, and autoencoder techniques described herein provide two advantages and improvements in implementations compared to related techniques. First, the techniques reduce the feature space of an application for incorporation into machine learning techniques. When a performance prediction model is run on a project, there are many model features available. However, project data tends to be sparse and not dense enough to train into a machine learning model. An example of a model feature may not have enough data (e.g., many columns of data but few rows of data), so the implementations described herein reduce the feature space to facilitate incorporation into machine learning techniques. Furthermore, projects under project management may take a long time to accelerate (e.g., one or two years), which causes sparseness of the data used in training machine learning techniques. By using the feature space reduction described herein, the data can be used to train a model to predict the appropriate KPIs. As a result, both the execution time and the accuracy of the model are improved. Second, the techniques can help reduce noise in the raw data and features. As a result, only the signal and a small amount of noise in the raw data and features are kept, which helps improve the accuracy of the model and generalize the model to unknown test data. Insight Discovery 120
[0054] Given data-derived characteristics and performance data, insights can be discovered by insight discovery 120 and used for decision making and downstream tasks. Depending on what data is used to discover insights about a project, there are two solution components: Self-profiling component 103 discovers insights about a project based on project characteristics. Performance monitoring component 104 discovers insights about a project based on project performance.
[0055] The self-profiling component 103 processes the project characteristics and discovers insights about the projects. Self-profiling is done by unsupervised learning, whereby a clustering algorithm can discover clusters of projects based on project similarities and assign each project to a cluster, while an anomaly detection algorithm can identify anomalies in a project based on the significant difference of the project from the rest of the projects and determine whether the project is an anomalous project.
[0056] In the implementation described herein, a framework for solving unsupervised learning tasks by supervised learning techniques is utilized. For each unsupervised learning task, multiple unsupervised models for model algorithms and parameter sets are constructed and evaluated. Based on the model evaluation results, the best model is selected as the final unsupervised model for the unsupervised learning task. Here, there are two learning schemes for unsupervised models by reusing one component in the framework described in PCT Application No. PCT / US2020 / 58311, filed October 30, 2020, “AUTOMATED REAL-TIME DETECTION, PREVENTION, AND PREVENTION OF RARE FAILURES IN INDUSTRIAL SYSTEM WITH UNLABELED SENSOR DATA,” the entire contents of which are incorporated herein by reference for all purposes.
[0057] FIG. 2 illustrates a framework component 210 for facilitating unsupervised machine learning with hyperparameter tuning according to an example implementation. For each unsupervised machine learning model algorithm in the unsupervised machine learning module 201, this component is used to find a parameter set that leads to a model that meets the desired selection criteria (e.g., the model with the best accuracy, the model with the lowest error, etc.), and such a model and parameter set are termed the "best" model and the "best" parameter set herein. Each parameter set is used to build an unsupervised model. The models are then evaluated using the ensemble supervised learning module 202, and the best model with a given parameter set is selected as the best unsupervised model 203. In the unsupervised machine learning with hyperparameter tuning module 210 of FIG. 2, the input is the features coming from the feature engineering module 102, and the output is the best unsupervised model 203.
[0058] Figure 3 shows an example of an iterative learning scheme according to an implementation example. The unsupervised machine learning with hyperparameter tuning module 210 of Figure 2 implements the iterative learning scheme shown in Figure 3. The flow of the iterative learning scheme is as follows.
[0059] First, an unsupervised model algorithm is selected and applied to the features coming from the feature engineering module 102. After the best unsupervised model for a given model algorithm is identified, the model is applied to the features to generate unsupervised outputs. The unsupervised outputs are then added to the features. This flow is repeated until a termination criterion is met. Depending on the desired implementation, the termination criterion may include, but is not limited to, a model evaluation metric meeting a predefined threshold or a predefined list of unsupervised model algorithms being used.
[0060] Therefore, the learning scheme as shown in FIG. 3 features iterative application of the unsupervised learning model, adding the output of the unsupervised model in the current round as features to construct the unsupervised model in the next round. As described below, there are several modified forms of the iterative learning scheme.
[0061] In one example of the modification, there can be various types of machine learning related to the intermediate unsupervised learning model algorithm. In this scheme, the "unsupervised machine learning module M with hyperparameter adjustment" in FIG. 3 has the same type of machine learning as the "best unsupervised model". In other words, they are both "clustering" algorithms or both "anomaly detection" algorithms.
[0062] The "unsupervised machine learning module X with hyperparameter adjustment" where 1 <= X < M can have the same or different types of machine learning as the "best unsupervised model". For example, when the "best unsupervised model" is of the clustering type, the "unsupervised machine learning module X with hyperparameter adjustment" where 1 <= X < M can be "clustering" or "anomaly detection" regardless of the type of machine learning of the "best unsupervised model". The reason is that the outputs of unsupervised models of various types of machine learning can help discover insights into features and help construct the best model of the desired type of machine learning.
[0063] In another example of the modification, there can be optimization of the sequence of the unsupervised model algorithm. In such an iterative learning scheme, the sequence of the unsupervised model algorithm can bring about changes in the final best unsupervised model. The number and order of the unsupervised learning model algorithms are optimized by the forward model selection algorithm.
[0064] To initialize the scheme, a set of unsupervised learning model algorithms is selected based on machine learning knowledge. Then, each unsupervised learning model algorithm is applied to the features coming from the feature engineering module 102. Then, the best unsupervised model with the best parameter set is obtained from each unsupervised learning model algorithm. The evaluation result is recorded as the global best evaluation result. The first module ("Unsupervised Machine Learning with Hyperparameter Tuning Module 1") is used, and then the output from the best model is added to the features.
[0065] After the above initialization, the scheme performs forward selection, which randomly selects another unsupervised learning model algorithm and applies it to the current features. Then, for the current unsupervised learning model algorithm, the best unsupervised model with the current best evaluation result is obtained. If the current best evaluation result is better than the global best evaluation result, the current model in the sequence is kept and the scheme appends the output of the best unsupervised model to the features, otherwise it drops the current model. The forward selection is repeated until the existing criteria is met.
[0066] Another example of a modification is multi-tasking. Given data, it may be necessary to identify both clusters of projects and anomalies of projects. The iterative learning scheme can be modified to complete those two tasks within one run. For this task, the last two modules are modified to be two types of unsupervised machine learning, one clustering and the other anomaly detection. The best models from the last two modules are used to model the two tasks.
[0067] Figure 4 shows an example of a recursive learning scheme according to one implementation example. The "Unsupervised Machine Learning with Hyperparameter Tuning Module" 210 in Figure 2 implements the recursive learning scheme shown in Figure 4. The workflow is as follows:
[0068] First, a set of unsupervised model algorithms is selected based on knowledge of machine learning. Each unsupervised model algorithm in the set is applied to the features coming from the feature engineering module 102. The best model is selected along with the best parameter set, and then this scheme generates the unsupervised output from the best model for each model algorithm. The unsupervised output from each unsupervised model algorithm is added to the features. This recursive flow is repeated until an end criterion is met. Depending on the desired implementation, the end criterion is not limited to just this, but can include, among other things, the model evaluation metric meeting a predetermined threshold, a predetermined number of rounds (N) being executed, and / or the evaluation results converging, i.e., the evaluation result of the current round being close to the evaluation result of the last round based on a predetermined proximity threshold.
[0069] This learning scheme recursively applies a set of unsupervised learning model algorithms to the features, adds the unsupervised model output of the current round to the features, and constructs the unsupervised model within the next round.
[0070] There are several modified forms of the recursive learning scheme.
[0071] In an example of a modified form of the recursive learning scheme, there can be various types of machine learning for the intermediate unsupervised learning model algorithms. In the last round N, the "unsupervised machine learning module X with hyperparameter tuning" where 1 <= X <= M has the same type of machine learning as the "best unsupervised model". For each round Y where 1 <= Y < N, the "unsupervised machine learning module with hyperparameter tuning" where 1 <= X <= M has the same or a different type of machine learning as the "best unsupervised model".
[0072] Another example of a variation of the recursive learning scheme can be multi-tasking. Given data, it may be necessary to identify both clusters of projects and anomalies of projects. The recursive learning scheme can be modified to complete those two tasks in one run. To this end, in the last round N, "Unsupervised Machine Learning Module X with Hyperparameter Tuning", where 1<=X<=M, has a mix of two machine types: clustering and anomaly detection. The best models of each machine learning type in the last round are used as the best models for the two tasks.
[0073] Another example of a variation of the recursive learning scheme may be the dropout of model algorithms. In each round, some model algorithms may be dropped out, and only the remaining model algorithms are used in that round. The dropout rate is used to control how many model algorithms are dropped out. For example, "dropout rate = 0.1" means that in each round, 10% of the model algorithms are randomly selected and dropped out, and the remaining 90% of the model algorithms are used in each round. Since the model algorithms are randomly selected and dropped out in each round, the model algorithms used in each round may be different. The dropout rate may be adjusted by hyperparameter tuning techniques. One approach is to enumerate multiple dropout rates and try each of them to see which dropout rate may result in the best model.
[0074] Using dropout solves potential overfitting problems and makes the final model more generalizable. In addition, using a subset of the model algorithms can help reduce run time.
[0075] Another example of a variation of the recursive learning scheme can be data sampling, where a subset of data for training the model can be used by data sampling techniques in each round. Sampling rate to control how much data is used in each round. For example, "sampling rate=0.6" means that 60% of the data is used in each round, and the remaining 40% of the data is not used in each round. Since data is randomly sampled in each round, the data used in each round can be different. Sampling rate can be adjusted by hyperparameter tuning techniques. One approach is to enumerate multiple sampling rates and try each of them to see which sampling rate can result in the best model.
[0076] Using data sampling solves potential overfitting problems and makes the final model more generalizable. Additionally, using a subset of the data to train the model can help reduce run time.
[0077] For performance monitoring 104, a project may be divided into several milestones. A manager or stakeholder may review the project's progress at each milestone and take action based on the project's performance at each milestone. The project's performance may be measured by various KPIs as described herein. Thus, monitoring the KPIs at each milestone may derive several key insights, which may help the manager or stakeholder to make strategic business decisions.
[0078] The implementations shown in Figures 3 and 4 can improve the performance of the model with respect to bias and variance.
[0079] Regarding removing bias, for the same data set, various model algorithms with various parameter sets may generate completely different outputs. Thus, model bias occurs. How to remove model bias, and how to distinguish which output is the best, and how to determine the final model to select have been problems in the related art. The proposed approach in the implementation example uses several model algorithms and introduces some learning logic to ensemble them to derive the final model.
[0080] Regarding removing variance, if the dataset is very noisy and has a lot of variance, it may be difficult to use the model algorithm to find the real signal, which may cause the model to perform poorly. The implementation described herein utilizes the model output from the current round as additional features to build the model in the next round. The intermediate outputs are extracted from the intermediate models and may be considered closer to the signal in the original data. Using them as additional features can highlight more of the signal when building the model for the next round. The features gradually converge to the signal in each round, so that the final model may perform well on such features.
[0081] Another advantage of the implementations described herein includes automation, i.e., model selection and ensembling are all done within an automated pipeline, whereas in related art this is done by manual evaluation and validation, which is time-consuming and error-prone, in contrast to the automated implementations described herein.
[0082] Furthermore, the data used to build the model is problem specific. Given a problem, expert knowledge can be used to identify what data may be useful. Given the data, data analysis such as correlation analysis can be performed to determine the relationship between the feature data and the object. Feature transformation and feature selection techniques may also be used to derive and select features useful for building the model.
[0083] As an example, the implementations of Figures 3 and 4 can be extended to Internet of Things (IoT) data, such as IoT insurance or other IoT systems. In an example involving IoT insurance, a project may include insurance against machine downtime, or guaranteeing machine uptime over a period of time. IoT data comes from sensors mounted on assets. Multiple sensors are used collectively to derive some insight about the asset, for example, whether there is an anomaly or failure of the asset. The implementations of Figures 3 and 4 can be used to derive insight about the asset, including clusters of values of multiple sensors and anomalies of the asset.
[0084] Although the examples of Figures 3 and 4 have been described with respect to unsupervised learning tasks, such implementations may be further extended to supervised learning tasks depending on the desired implementation.
[0085] Figure 5 shows a KPI transition network according to one implementation example. In this network, KPIs for each project are categorized as "good", "fair" or "poor". For each KPI, the transition probability from one milestone to the next is captured and aggregated based on historical projects. One approach is to count the number of times a category in one milestone transitions to a category in the next milestone, and use that ratio as the transition probability from a category in one milestone to a category in the next milestone. This approach is done within each KPI.
[0086] For example, for KPI1, assume there are 10 historical projects in the "good" category in round 1. Of those 10 projects, 7 projects are in the "good" category in round 2, 2 projects are in the "fair" category in round 2, and 1 project is in the "poor" category in round 2. So the results are as follows: Probability of transitioning from "Good" (Milestone 1) to "Good" (Milestone 2) is 7 / 10 = 0.7 Probability of transitioning from "good" (Milestone 1) to "fair" (Milestone 2) is 2 / 10 = 0.2 Probability of transitioning from "good" (Milestone 1) to "bad" (Milestone 2) is 1 / 10 = 0.1
[0087] Given a new project and its category at the milestone, the transition graph can be applied to ascertain the possibility of transitioning to a different category at the next milestone. The transition probabilities are used as estimates of the category for KPIs and to determine business decisions accordingly. Such probabilities are considered as prior probabilities in terms of Bayesian networks. There are several variations on the transition networks described.
[0088] In a first variant, depending on the desired implementation, numerical KPI measurements can be used. For example, there can be five bins for each KIP: 0-20%, 20%-40%, 40%-60%, 60%-80%, 80%-100%. Each KPI value can be assigned to one of the bins and the transition network can be constructed accordingly.
[0089] In a second variant, a transition from one KPI in one milestone to another KPI in the next milestone can be promoted. Such a transition is a KPI crossing transition. This transition is useful when some KPIs are correlated and the category of one KPI in one milestone is related to the category of the correlated KPI in the next milestone.
[0090] In a third variant, there can also be cross-milestone transitions. Transitions from one milestone to another can be accelerated. For example, a transition can occur from milestone 1 to milestone 3. This is useful when the category of a KPI at an earlier milestone correlates with the category of a KPI at a later milestone. This helps project stakeholders or managers to get an estimate of the performance of several milestones in the future.
[0091] FIG. 6 illustrates a solution for deriving or predicting KPIs at a later milestone in a tabular format based on KPIs at a previous milestone according to an implementation example. The implementation example may predict multiple categories of KPIs at each milestone. For any given milestone T, a multi-task machine learning classification model may be built for all KPIs at this milestone. The features are the categories of each KPI at each milestone (milestone 1 to milestone T-1) of each project in the historical data. The targets are the categories of KPIs at milestone T, i.e., the three classes "good", "fair", and "poor". It should be noted that only performance KPI data is used as features and targets to build the machine learning model. As shown in FIG. 6, all KPIs may be used as features at milestones 1 to T-1, and KPIs at milestone T may be used as targets. Given a new project at milestone T-1, the model may be applied to predict the categories of all KPIs at milestone T.
[0092] This is a multi-task model in that multiple targets (i.e., multiple KPIs at milestone T) need to be predicted within one model. Multiple learning tasks are solved simultaneously, leveraging commonalities and differences between the tasks. By jointly learning task-specific models, this method can result in improved learning efficiency and prediction accuracy of the task-specific models compared to training the models separately.
[0093] Multi-task learning works better than single-task learning because the regularization that comes from requiring algorithms to perform well on related tasks can be better than regularization that prevents overfitting by penalizing all complexity uniformly. In our problem, the KPIs at milestone T are likely to be correlated, and the tasks of predicting the KPIs share significant commonality and are typically slightly undersampled. On the other hand, for unrelated KPIs, multi-task learning is also beneficial for learning unrelated tasks.
[0094] Figure 7 shows an example implementation of performance prediction 105 from Figure 1 that can predict project performance at each milestone and identify the root causes of each prediction. Machine learning techniques are deployed to predict project performance by milestone and explain those predictions.
[0095] For each milestone of the project, the implementation predicts the performance of the project at that milestone based on all available data prior to that milestone. First, data 101 (e.g., project data, employee data, client data) as described herein is collected. Feature engineering 102 is used to generate features 700 from the data. Feature engineering techniques include identifying useful attributes in the data and using them as features based directly on expert knowledge, manual feature transformation and derivation, based on expert knowledge and data analysis and automatic feature derivation by the techniques described herein. The generated features may include original features and derived features. Some features may be used directly as features without data transformation, they are original features. Other features are derived from the data with data transformation or from a feature derivation process, they are derived features. One way to derive features based on expert knowledge and data analysis is to aggregate performance data of each individual based on the performance of the projects in which the individual is working. Similarly, aggregation may be done by team, organization, and client. Additionally, automatic feature derivation techniques include PCA, ZCA, and autoencoders that help reduce the feature space, reduce noise, and preserve key signals in the data.
[0096] For project self-profiling 103, the self-profiling techniques described herein are applied to features from feature engineering module 102 to derive hidden attributes including project clusters and project anomalies for rendering as self-profiling results 701. Such hidden attributes can be used to predict the performance of the project at the next milestone.
[0097] For performance monitoring 104, the performance monitoring techniques described herein are applied to the performance KPI data to derive transition probabilities (based on the transition network of FIG. 5) and / or probabilities (based on the performance prediction model of FIG. 6) of each KPI category at the next milestone, as provided as monitoring results 702. Such probabilities may be used to build a model for predicting performance at the next milestone.
[0098] To build a supervised machine learning model, features and targets need to be prepared. Thus, in FIG. 7, the generated features from features 700, derived clusters and derived anomalies from self-profiling results 701, and probabilities of key performance indicator values from monitoring results 702 are combined into a supervised machine learning model 703 (as "features" in the supervised machine learning model) to build a supervised machine learning model for predicting project performance at each milestone (as "targets" in the supervised machine learning model). If performance is measured by categorical values such as "good", "fair", or "poor", a classification model may be applied, and if performance is measured by numerical values, a regression model 703 may be applied. Several model algorithms may be used, including but not limited to linear regression, random forest, gradient boosting, neural networks, etc. Then, the hyperparameters of the selected model algorithm may be tuned to select the best model. In addition, feature selection may be applied to select features that predict project performance.
[0099] Once the supervised machine learning model is constructed, it can be used to predict the performance of a project. Predicted performance 704 shows the project's performance predicted from supervised machine learning model 703.
[0100] Given the predicted performance 704, root cause analysis may be performed to identify key contributing factors for the predicted project performance. Root cause analysis 705 is used to identify the underlying root causes for each predicted project performance. Based on the expertise related to the root causes, stakeholders or managers can make business decisions. The root causes 706 may be linked to normative actions (e.g., based on expertise), and the process for obtaining normative actions given the root causes 706 may be automated.
[0101] Identifying the root causes 706 of a prediction corresponds to interpreting the prediction within the realm of machine learning, and several techniques and tools exist for such a task. For example, "ELI5" is an explainable artificial intelligence (AI) package that can help identify key features that lead to a prediction. Key features may have a positive impact on the prediction or a negative impact on the prediction. ELI5 may output the top P positive key features and the top Q negative key features.
[0102] For milestone-based performance optimization, project stakeholders or managers need to identify the best project team to optimize project performance by incorporating constraints for each given project, team, client, etc. The implementation utilizes a multi-level, multi-dimensional collaborative filtering approach to solve the project performance optimization problem.
[0103] Collaborative filtering is the process of filtering for information or patterns using techniques that involve collaboration among multiple agents, viewpoints, data sources, etc. A popular application of collaborative filtering is a recommender system where items can be recommended to users based on similar users' interest in the items. Traditionally, a matrix with "users" as rows and "items" as columns can be constructed based on historical data, and collaborative techniques such as similarity-based techniques and matrix factorization can be applied to fill missing values in the matrix. Here, a multi-level collaborative filtering technique is introduced to solve a project performance optimization problem.
[0104] Figure 8 shows an example implementation of performance optimization 106 of Figure 1 with a multi-level multi-dimensional collaborative filtering matrix. Given past project performance data, a matrix can be formed as shown in Figure 8 as follows:
[0105] The rows are employee-related data at different levels, for different dimensions, or for different groups of employees. According to one implementation, the employee-related data may include individual employee IDs or names, teams in an organizational chart, organizations in an organizational chart, clusters of employees based on self-profiling of projects, employee attributes, etc.
[0106] The columns are project related data at different levels, of different dimensions, or for different project groups. According to a desired implementation, the columns include individual projects, clusters of projects based on self-profiling of projects, attributes of projects, etc.
[0107] The values are performance scores as described herein, for example the rate of return from a performance KPI may be a good candidate for measuring the overall performance of a project.
[0108] The matrix has missing values and a training process is used to fill in the missing values by using collaborative filtering techniques including algorithms based on similarities between items and matrix factorization algorithms.
[0109] In algorithms based on similarity between items, the algorithm first builds a model by computing similarity scores between all pairs of item vectors. The similarity function can be of various forms, such as correlation between item vectors or cosines of item vectors. The algorithm then uses the most similar items to the user's already rated items to generate a list of recommendations for the user. The intuition behind this recommendation is "users who rate item X as highly as you do also tend to rate item Y highly, and you should try item Y since you haven't rated it yet."
[0110] In a matrix factorization algorithm, a matrix (with some existing values and many missing values) is factored into two vectors: user latent factors (column vectors, which can also be thought of as coefficients in a latent model) and item latent factors (row factors, which can also be thought of as features of items in a latent model). The two vectors can be multiplied to make predictions and fill in missing values in the matrix. The factorization process can be done by gradient descent optimization, which takes one fixed vector and improves the remaining vector each time.
[0111] Once the matrix is constructed, it can be used to generate an optimal team for a new project. Given a new project, it is necessary to find the employees, teams, organizations, clusters of employees, and employee attributes that are best suited for the project to achieve the best performance. An implementation can utilize a population model inference process to identify the project team at the employee group level, which can be individual employees, teams, organizations, clusters of employees, and employees with the same attributes.
[0112] First, given a new project, the process finds all matching columns based on the project information. For the employee group level of interest, the process identifies all employee groups at the employee group level. For each employee group, the process finds all matching rows. For each employee group, the process gets all performance scores for all matching rows and columns.
[0113] The process then aggregates the performance scores. The aggregation method can be a simple average of the performance scores or a weighted average of the performance scores, which can include dividing the average of the performance scores by the number of employees contained in the matching employee group row. The process then ranks all the aggregated performance scores for the employee groups within the employee group level and selects the employee group with the highest performance score for the project.
[0114] In the related art, traditional collaborative filtering techniques tend to fail because there is a very limited number of relationships between individual employees and individual projects. As a result, a matrix with individual employees as rows and individual projects as columns becomes very sparse. Collaborative filtering techniques do not work well for sparse matrices. An implementation introduces employee groups and project groups into the matrix to obtain a denser matrix. Collaborative filtering techniques work well for matrices with employee groups as rows and project groups as columns.
[0115] By incorporating various employee group and project group levels in one matrix, there can be some overlapping employees of employee groups from various employee group levels. As a result, the performance scores of employee groups with overlapping employees are correlated, and such correlation can be useful for collaborative filtering, since those employees have similar involvement with items (projects in this case). Data from various employee group levels and project group levels can help each other in filling the scores in the matrix.
[0116] The population model inference process helps consider various aspects of employees and projects and aggregate them together into a single performance score.
[0117] Sometimes project stakeholders need to identify project teams per milestone depending on human resource availability, strengths and interests, project duration, etc. For that, the implementation makes collaborative filtering milestone-based. Instead of using performance scores over the entire project duration to fill the matrix, performance scores for each milestone are used to fill the matrix. So there can be one matrix per milestone or more depending on the desired implementation.
[0118] The implementations may be incorporated into various scenarios. For example, the implementations may facilitate an end-to-end solution. As a solution suite for project insight discovery and project performance modeling, project self-profiling, performance monitoring, performance prediction, and performance optimization may be provided. This end-to-end solution may be provided as an Analytics Solution Core Suite as part of a Solution Core product.
[0119] Project self-profiling may be provided as part of the Solution Core product as an Analysis Solution Core.
[0120] Performance Monitoring may be offered as part of the Solution Core product as Analytics Solution Core.
[0121] Performance prediction may be provided as an Analytics Solution Core as part of the Solution Core product.
[0122] Project Performance Optimization may be offered as an Analytics Solution Core as part of the Solution Core product.
[0123] Implementations can be incorporated into a standalone machine learning library. The iterative and recursive learning schemes for self-profiling can be provided as a standalone machine learning library to aid in self-profiling and insight discovery.
[0124] FIG. 9 illustrates an example computing environment having an example computer device suitable for use with some implementations.
[0125] A computing device 905 in computing environment 900 may include one or more processing units, cores, or processors 910, memory 915 (e.g., RAM, ROM, etc.), internal storage 920 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interface 925, any of which may be connected to a communication mechanism or bus 930 for communicating information or may be embedded in the computing device 905. The I / O interface 925 may also be configured to receive images from a camera or provide images to a projector or display depending on the desired implementation.
[0126] The computing device 905 may be communicatively connected to an input / user interface 935 and an output device / interface 940. Either or both of the input / user interface 935 and the output device / interface 940 may be wired or wireless interfaces and may be removable. The input / user interface 935 may include any physical or virtual device, component, sensor, or interface that can be used to provide input (e.g., buttons, touch screen interface, keyboard, pointing / cursor control, microphone, camera, Braille, motion sensor, optical reader, etc.). The output device / interface 940 may include a display, television, monitor, printer, speaker, Braille, etc. In some implementations, the input / user interface 935 and the output device / interface 940 may be embedded in the computing device 905 or may be physically connected to the computing device 905. In other implementations, other computing devices may function as or provide the functionality of the input / user interface 935 and the output device / interface 940 for the computing device 905.
[0127] Examples of computing devices 905 may include, but are not limited to, highly mobile devices (e.g., smart phones, devices in vehicles and other machines, devices carried by humans and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions, radios with one or more processors embedded and / or coupled thereto, etc.).
[0128] Computing device 905 may be communicatively connected (e.g., via I / O interface 925) to external storage 945 and a network 950 for communicating with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 905, or any connected computing device, may function as, provide services to, or be referred to as, a server, a client, a thin server, a general purpose machine, a special purpose machine, or another label.
[0129] I / O interface 925 may include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMax, modem, cellular network protocols, etc.) to communicate information to and from at least all connected components, devices, and networks in computing environment 900. Network 950 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).
[0130] The computing device 905 may use and / or communicate using computer usable or computer readable media, including transitory and non-transitory media. Transitory media include transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, etc. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.
[0131] In some examples of computing environments, the computing device 905 may be used to implement techniques, methods, applications, processes, or computer-executable instructions. The computer-executable instructions may be obtained from a transitory medium or stored on and obtained from a non-transitory medium. The executable instructions may be from one or more of any programming language, scripting language, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).
[0132] The processor 910 may execute under any operating system (OS) (not shown) in a native or virtual environment. One or more applications may be implemented, including a logic unit 960, an application programming interface (API) unit 965, an input unit 970, an output unit 975, and an inter-unit communication mechanism 995 for the various units to communicate with each other, with the OS, and with other applications (not shown). The described units and elements may be modified in design, function, configuration, or implementation and are not limited to the provided description.
[0133] In some implementations, when information or instructions to execute are received by the API unit 965, it may be communicated to one or more other units (e.g., logic unit 960, input unit 970, output unit 975). In some implementations described above, logic unit 960 may be configured in some instances to control information flow between units and direct services provided by API unit 965, input unit 970, output unit 975. For example, the flow of one or more processes or implementations may be controlled by logic unit 960 alone or in combination with API unit 965. Input unit 970 may be configured to obtain inputs for calculations described in the implementations, and output unit 975 may be configured to provide outputs based on the calculations described in the implementations.
[0134] The processor 910 may be configured to execute a method for facilitating project management as described in the implementations herein, which may include, for a project input having project data and employee data, as shown in FIG. 7, performing feature extraction on the project data and employee data to generate features, running a self-profiling algorithm consisting of unsupervised machine learning on the generated features to derive clusters and anomalies for the project, running a performance monitoring process on the generated features to determine probabilities of key performance indicator values at milestones, and running a supervised machine learning model on the generated features, the derived clusters, the derived anomalies, and the probabilities of key performance indicator values at milestones to generate a predicted performance value for the project.
[0135] In the example implementation shown in FIG. 2, the processor 910 may be configured to execute a self-profiling algorithm comprised of unsupervised machine learning on the generated features to derive clusters and anomalies for the project by performing unsupervised machine learning to generate unsupervised machine learning models based on the generated features, performing supervised machine learning on results from each of the unsupervised machine learning models to generate supervised ensemble machine learning models, each of the supervised ensemble machine learning models corresponding to each of the unsupervised machine learning models, and selecting certain of the unsupervised machine learning models as models configured to derive clusters and anomalies based on an evaluation of results of the unsupervised machine learning models against predictions generated by the supervised ensemble machine learning models.
[0136] In the example implementation shown in FIG. 3, the processor 910 may be configured to execute a self-profiling algorithm comprised of unsupervised machine learning on the generated features to derive clusters and anomalies for the project by executing each unsupervised machine learning model algorithm from the set of unsupervised learning model algorithms on the generated features, determining one of the unsupervised machine learning models having associated parameter sets for each unsupervised machine learning model algorithm that meets a selection criterion, and determining an unsupervised machine learning model for deployment across the set of unsupervised machine learning model algorithms from one of the unsupervised machine learning models from each unsupervised machine learning model algorithm that meets the selection criterion.
[0137] In the implementation example shown in FIG. 3, the processor 910 may further introduce an unsupervised machine learning model for introduction, and during the introduction of the unsupervised model for introduction, the processor 910 may further execute the following methods: applying the unsupervised machine learning model for introduction to the generated features to generate unsupervised outputs, appending the unsupervised outputs to the features to obtain extended features, randomly selecting another unsupervised learning model algorithm from the set of unsupervised machine learning model algorithms, training the randomly selected unsupervised learning model algorithm based on the extended features to find another unsupervised machine learning model having another associated parameter set that satisfies a selection criterion, and replacing the introduced unsupervised learning model for introduction with the other unsupervised learning model for introduction with respect to the other unsupervised learning model generated from the randomly selected unsupervised learning model algorithm having a better evaluation than the introduced unsupervised learning model for introduction.
[0138] As shown in FIG. 4, the processor 910 may be configured to execute a self-profiling algorithm comprised of unsupervised machine learning on the generated features to derive clusters and anomalies for the project by a) applying a set of unsupervised machine learning model algorithms to the generated features to generate an unsupervised machine learning model for each of the unsupervised machine learning algorithms, b) appending unsupervised output from the unsupervised machine learning models to the features, and c) repeating steps a) and b) until a termination criterion is met.
[0139] As shown in FIG. 5, the processor 910 may be configured to perform a performance monitoring process on the generated features to determine a probability of a key performance indicator value at the milestone by generating a transition network from historical projects that relates transitions between a plurality of first key performance indicators at a first milestone to a plurality of second key performance indicators at a second milestone, where the probability of the transition is determined based on the number of times the plurality of first key performance indicators at the first milestone transition to a second key performance indicator at the second milestone.
[0140] As shown in Figures 5 and 6, the processor 910 may be configured to perform a performance monitoring process on the generated features to determine a probability of a key performance indicator value at the milestone by generating a multi-task supervised machine learning model for predicting a key performance indicator value at a milestone based on a key performance indicator value at a previous milestone, where the probability of the key performance indicator value is one of a category when a classification model is used as the multi-task supervised machine learning model or a numerical score when a regression model is used as the multi-task supervised machine learning model, and the key performance indicator value at the milestone is predicted simultaneously with the multi-task supervised machine learning model.
[0141] As shown in FIG. 8 , the processor 910 may be configured to execute a method including generating a matrix from the performance data of the projects, the matrix being comprised of rows of employee-related data and columns of project-related data across various dimensions and including values indicative of performance scores derived from the performance data of the projects; for missing values in the matrix, generating missing values in the matrix from a training process; for an input of a new project, identifying rows and columns that match the employee-related data in the new project and the project-related data of the new project, aggregating values corresponding to the identified rows and columns, ranking employee groups from the employee-related data according to the aggregated values, and selecting the employee group from the employee-related data having the highest rank for execution of the new project.
[0142] As shown in FIG. 7, the processor 910 may be configured to: run a supervised machine learning model on the generated features, derived clusters, derived anomalies, and probabilities of key performance indicator values at the milestones to generate a predicted performance value for the project; run a self-profiling algorithm consisting of unsupervised machine learning on the generated features to derive clusters and anomalies for the project; run a performance monitoring process on the generated features to determine probabilities of key performance indicator values at the milestones; and run a supervised machine learning model on the generated features, derived clusters, derived anomalies, and probabilities of key performance indicator values at the milestones to generate a predicted performance value for the project.
[0143] Implementations may be extended to IoT data fields such as IoT insurance, as described in Figure 3. In such implementations, the processor 910 may be configured to execute a computer-implemented method for executing a self-profiling algorithm comprised of unsupervised machine learning on the generated features to derive clusters and anomalies of Internet of Things (IoT) data, which may include executing each unsupervised machine learning model algorithm from a set of unsupervised learning model algorithms on the generated features, determining for each unsupervised machine learning model algorithm a best of the unsupervised machine learning models having an associated best parameter set, and determining a best unsupervised model across the set of unsupervised machine learning model algorithms from the best of the unsupervised machine learning models for each unsupervised machine learning model algorithm.
[0144] Some portions of the detailed description have been presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the substance of their innovations to others skilled in the art. An algorithm is a sequence of defined steps leading to a desired end state or result. In one implementation, the steps performed require physical manipulations of tangible quantities to achieve a tangible result.
[0145] Unless otherwise specified, as is apparent from the description, the description utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," and the like throughout the description will be understood to include the actions and processes of a computer system or other information processing device that manipulates and converts data represented as physical (electronic) quantities in the computer system's registers and memory to other data similarly represented as physical quantities in the computer system's memory or registers, or other information storage, transmission, or display device.
[0146] The implementations may also relate to an apparatus for performing the operations of the present specification. The apparatus may be specially constructed for the required purposes, or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored in a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium may include tangible media, such as, but not limited to, optical disks, magnetic disks, read-only memories, random access memories, solid-state devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. The computer-readable signal medium may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. The computer programs may include pure software implementations that include instructions to perform the operations of the desired implementation.
[0147] Various general-purpose systems may be used with the programs and modules according to the examples herein, or it may prove convenient to construct more specialized apparatus to perform the desired method steps. Additionally, the implementations are not described with reference to a particular programming language. It will be understood that a variety of programming languages may be used to implement the techniques of the implementations described herein. Instructions in a programming language may be executed by one or more processing devices, such as a central processing unit (CPU), processor, or controller.
[0148] As known in the art, the operations described above may be performed by hardware, software, or some combination of software and hardware. Various aspects of the implementations may be implemented using circuits and logic (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software) that, when executed by a processor, cause the processor to perform a method for performing the implementation of the present application. Furthermore, some implementations of the present application may be performed exclusively in hardware, while other implementations may be performed exclusively in software. Furthermore, the various functions described may be performed in a single unit or distributed among several components in any number of ways. When performed by software, the method may be executed by a processor, such as a general purpose computer, based on instructions stored on a computer-readable medium. If desired, the instructions may be stored on the medium in compressed and / or encrypted format.
[0149] Moreover, other implementations of the present application will be apparent to those skilled in the art upon consideration of the specification and practice of the techniques of the present application. Various aspects and / or components of the described implementations may be used alone or in any combination. It is intended that the specification and implementations be considered as examples only, with a true scope and spirit of the present application being indicated by the following claims.
Claims
1. 1. A computer-implemented method for facilitating project management, comprising: for a project input including project data and employee data, performing feature extraction on the project data and the employee data to generate features; running a self-profiling algorithm comprised of unsupervised machine learning on the generated features to derive clusters and anomalies for the project; performing a performance monitoring process on the generated features to determine probabilities of key performance indicator values at milestones; and running a supervised machine learning model on the generated features, the derived clusters, the derived anomalies, and the probabilities of the key performance indicator values at the milestones to generate a predicted performance value for the project. Including, said executing said performance monitoring process on said generated features to determine said probability of a key performance indicator value at a milestone; generating a transition network from the historical project relating transitions between a first plurality of key performance indicators at a first milestone to a second plurality of key performance indicators at a second milestone; Including, the probability of the transition is determined based on a number of times the plurality of first key performance indicators of the first milestone transition to the second key performance indicator of the second milestone. A computer-implemented method.
2. 1. A computer-implemented method for facilitating project management, comprising: for a project input including project data and employee data, performing feature extraction on the project data and the employee data to generate features; running a self-profiling algorithm comprised of unsupervised machine learning on the generated features to derive clusters and anomalies for the project; performing a performance monitoring process on the generated features to determine probabilities of key performance indicator values at milestones; and running a supervised machine learning model on the generated features, the derived clusters, the derived anomalies, and the probabilities of the key performance indicator values at the milestones to generate a predicted performance value for the project. Including, said performing a performance monitoring process on said generated features to determine said probability of said key performance indicator value at said milestone; generating a multi-task supervised machine learning model for predicting the key performance indicator value at the milestone based on the key performance indicator value at a previous milestone; Including, the probability of the key performance indicator value is one of a category when a classification model is used as the multi-task supervised machine learning model, or a numerical score when a regression model is used as the multi-task supervised machine learning model; The key performance indicator values at the milestones are simultaneously predicted. A computer-implemented method.
3. generating a matrix from project performance data, the matrix being comprised of rows of employee-related data and columns of project-related data across various dimensions and including values indicative of performance scores derived from the project performance data; for any missing values in the matrix, generating the missing values in the matrix from a training process; Regarding entering a new project, identifying the rows and columns that match the employee-related data in the new project and the project-related data of the new project; aggregating the values corresponding to the identified rows and columns; ranking employee groups from the employee-related data according to the aggregated values; and selecting a group of employees from said employee-related data having the highest ranking with respect to carrying out said new project; A computer-implemented method comprising:
4. running a supervised machine learning model on the generated features, the derived clusters, the derived anomalies, and the probabilities of key performance indicator values at milestones to generate a predicted performance value for the project; running a self-profiling algorithm comprised of unsupervised machine learning on the generated features to derive clusters and anomalies for the project; running a performance monitoring process on the generated features to determine probabilities of the key performance indicator values at milestones; running a supervised machine learning model on the generated features, the derived clusters, the derived anomalies, and the probabilities of the key performance indicator values at the milestones to generate a predicted performance of the project. The method according to claim 3.
5. said performing said performance monitoring process on said generated features to determine said probability of said key performance indicator value at said milestone; generating a transition network from the historical project relating transitions between a first plurality of key performance indicators at a first milestone to a second plurality of key performance indicators at a second milestone; Including, the probability of the transition is determined based on a number of times the plurality of first key performance indicators of the first milestone transition to the second key performance indicator of the second milestone.
5. The computer-implemented method of claim 4.
6. said executing a performance monitoring process on said generated features to determine probabilities of key performance indicator values at milestones; generating a multi-task supervised machine learning model for predicting the key performance indicator value at the milestone based on the key performance indicator value at a previous milestone; Including, the probability of the key performance indicator value is one of a category when a classification model is used as the multi-task supervised machine learning model, or a numerical score when a regression model is used as the multi-task supervised machine learning model; The key performance indicator values at the milestones are simultaneously predicted.
5. The computer-implemented method of claim 4.
Citation Information
Patent Citations
System and method for supporting organization management
JP2007042059A
Project management support system
JP2014203228A
Software defect prediction device, software defect prediction method, and software defect prediction program
JP2016024477A
Project analysis device and its program
JP2019148874A
Predicting performance of a software project
US20120317536A1
