Techniques for Metric Data-Driven Correlation

Abstract

Described herein are techniques for identifying highly relevant content for a user to view in the form of KPI cards and providing relevant views to the user automatically or through suggestions. KPIs of highest practical and statistical significance are provided when the user accesses the user interface. In some embodiments, as the user views a KPI, other relevant KPIs may be provided for the user to view as suggestions. Additionally, in some embodiments, the user may be provided with significant KPIs based on anomaly detection, and an explanation of the anomaly and suggestions for remediating any issues may be provided to the user. Highly informative content may be surfaced through the use of advanced machine learning algorithms described herein.

Description

Technical Field

[0001] Reference to Related Applications This application claims the benefit and priority of Provisional Patent Application No. 62 / 855,218, filed on May 31, 2019, entitled "TECHNIQUES FOR DATA-DRIVEN CORRELATION OF METRICS", and Provisional Patent Application No. 62 / 737,518, filed on September 27, 2018, entitled "TECHNIQUES FOR DATA-DRIVEN CORRELATION OF METRICS", the entire contents of each of which are hereby incorporated by reference in their entirety for all purposes.

Background Art

[0002] Background Metrics (also referred to herein as key performance indicators ("KPIs")) can provide useful information regarding a company's operating capabilities. It can be difficult to identify which metrics are most useful to a user when making decisions. Information that is relevant to a user can change or stagnate quickly. However, a user who only periodically looks at a specific metric, especially if the user only periodically looks at metrics that are stagnating, may not notice significant changes or changes that contain useful information.

Summary of the Invention

[0003] Summary The technology described herein provides useful and statistically useful information to a user by providing information - providing, relevant, and personalized metrics to that particular user. A system consisting of one or more computers can be configured to perform certain operations or actions by having software, firmware, hardware, or combinations thereof installed on the system and causing the system to perform actions during operation. One or more computer programs can be configured to perform certain operations or actions by including instructions that, when executed by a data - processing device, cause the data - processing device to perform actions. One general aspect includes a computer - implemented method for identifying and presenting to a user highly information - providing content, including generating a model for each metric indicator of a plurality of metric indicators of an enterprise, the model having each of a plurality of attributes of the metric indicator as an independent variable of the model. For each metric indicator, based on the model, at least one attribute having the largest entropy - change contribution among the plurality of attributes is selected. A set of metric indicators is generated based on selecting a subset of the plurality of metric indicators, the subset of the plurality of metric indicators being selected at least in part based on having metrics with the largest statistically significant change. A graphical representation for each of the subset of the plurality of metric indicators is generated, each graphical representation of the subset of the metric indicators including at least one of the plurality of attributes. The graphical representation is provided to the user's device for presentation to the user. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0004] The implementation example may include one or more of the following features. Optionally, selecting a subset of multiple metric indicators involves obtaining a previous time-series model-based forecast of the current metric indicator, determining the current value of the metric indicator, and selecting the metric indicator if the previous forecast (expected value) is significantly statistically different from the current value. Optionally, generating an explanation of the metric with the largest entropy change for a subset of multiple metric indicators involves applying the current value of the metric indicator to a model along with the values of other independent metrics or variables to generate the partial derivative (rate of change) of each of the attributes (independent variables) in the model, and using computational techniques including but not limited to local interpretable model explanations and Shapley Additive Explanations to order the generated derivatives by size that are statistically significant, and selecting each subset of attributes based on how large its contribution is to the change in the metric. Optionally, the method may include recording the user's actions within the user interface and generating user preference information, usage frequency, and the metrics viewed before and after each metric for at least one metric indicator based on the recorded actions. Optionally, a set of metric indicators includes at least one metric indicator within the set. Optionally, generating a graphical display for each of the subsets of metric indicators includes generating a metric card that includes at least one visual depiction of at least one attribute. Optionally, the usage behavior of multiple users using the user interface is tracked and used to provide suggestions for metrics of interest based on what other users like the current user have viewed in the past. Optionally, trend data for each of multiple metrics based on the usage behavior of multiple users is generated. Optionally, a graphical trend card including a visual depiction of the trend data is generated.Optionally, the graphical trend card is provided in the form of a proposal and recommendation of a metric of interest to one or more of a plurality of users in the user interface. Implementations of the described technology may include hardware, a method or process, or computer software on a computer-accessible medium.

Brief Description of the Drawings

[0005]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Best Mode for Carrying Out the Invention

[0006] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular embodiments of the invention. It will be apparent, however, that the various embodiments may be practiced without these specific details. The figures and the description are not intended to be limiting. The term "exemplary" as used herein is intended to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments or designs.

[0007] Described herein are methods and systems for providing suggestions of metric indicators (also referred to as key performance indicators (“KPIs”)) for viewing in a user interface. Also described are techniques for providing highly informational content to a user without the user having to request a specific KPI. The user interface can provide a dashboard (also referred to as a cockpit) that can include visual depictions of metric indicators (e.g., business metric indicators or key performance indicators). The user interface can be provided by a computer system such as computer system 1200 of FIG. 12 , cloud infrastructure system 1102 of FIG. 11 , or server 1012 of FIG. 10 . The user interface can be provided to and useful to a user for viewing the metric indicators or the data underlying the metric indicators. Metric indicators can be data aggregations or snapshot views of the underlying data or time series trends created from that data with statistical transformations including, but not limited to, mean, median, standard deviation, quartiles, etc. For example, a metric indicator can provide trend data (e.g., sales have increased 10% over the past 60 days), snapshot information (e.g., year-to-date revenue is $10 million), etc. Business metrics can include values such as headcount, revenue, attrition, inventory, etc. Metric indicators (e.g., key performance indicators) can be generated based on metrics. For example, a year-to-date revenue metric indicator can provide information about year-to-date revenue. Metric indicators can provide trend or historical information that helps a user understand a metric in some context. For example, a daily revenue value may not be as useful to a user as a quarterly revenue value.

[0008] As used herein, metric, metric indicator, variable, metric card, KPI, and KPI card are used interchangeably to describe data provided with respect to a metric and / or data associated with a metric in a user interface. For example, KPI card 220a is a KPI card that indicates information regarding the headcount metric. A collection of KPI cards is called a set. For example, KPI cards 220a, 220b, 220c, 220d, 220e, 220f, 220g, and 215 are a set of KPI cards that are displayed to a user as shown in FIG. 2A. In some embodiments, the KPI cards within a set are related. In some embodiments, a user may select KPI cards that may or may not be related for display in a set in that there may or may not be metadata within the KPI cards that relate them to each other.

[0009] The user interfaces described can include, for example, user interfaces used by officers and teams of officers. Officers can include, for example, a chief financial officer (CFO), a chief technical officer (CTO), a chief human resources officer (CHRO), a chief executive officer (CEO), a line of business manager, a business analyst, and the like. As described herein, the user interface can be referred to as a dashboard. Optionally, the user interface can be used by any user.

[0010] When the described user interface is first initiated such that the business is provided as a cloud application, site installation, or any other suitable implementation, the user interface for any given user can be generated based on the user's type, area of interest, role, and responsibilities. For example, a CHRO may have a different view (e.g., different metric indicators shown) than a CEO. A CHRO may be more interested in employee data such as headcount, headcount reduction, diversity, etc. However, a CEO may be more interested in revenue, inventory, production, etc. Thus, an immediately available implementation can provide metric indicators that are more likely to be useful to the user based on the user's position or role within the business. Optionally, when installing the application, the user may select an initial dashboard configuration from several immediately available default configurations including those for a CEO, CHRO, CFO, CTO, etc.

[0011] With each user's use of the user interface, the computer system can track and identify each user's usage patterns. The patterns can include initial browsing habits, sequential browsing habits, habits that can be identified as changing based on data values, etc. For example, a CHRO may develop a pattern of opening the user interface, browsing the headcount metric indicator, and if the headcount metric indicator indicates a high headcount, then browsing the hiring metric indicator. However, if the headcount metric indicator is low, the CHRO may browse the headcount reduction metric indicator and view the headcount reduction metric. These browsing habits and their order can be analyzed and remembered. The details of this episodic memory storage are described in more detail with respect to FIG. 13.

[0012] Once a user's behavioral patterns are identified, the computer system may additionally use current data values or trends to identify information that may be useful to the user based on the user's usage patterns. For example, a CHRO may be given a suggestion to view headcount metrics upon login because the headcount metrics are low. Optionally, the suggestion may include a notification along with the suggestion. For example, the suggestion may appear in the user interface as a pop-up containing a suggestion and notification such as, "Headcount has decreased since last month and is at a monthly low point. Would you like to see the headcount metrics?" The suggestion and notification wording is clickable so that the user can click the text to view the headcount metrics within the user interface. In some embodiments, relevant data is provided to the user without requiring a user preference simply because it is statistically significant or because it is relevant for a particular domain.

[0013] In addition, by using the usage behavior of all users, input information from users, and the behavior of metric indicators over time, the relationships between metric indicators can be identified. For example, the behavior of a CHRO who views a headcount decrease metric indicator shortly after (within a threshold period) the headcount metric indicator can cause the computer system to identify the relationship between the headcount decrease metric indicator and the headcount metric indicator. Similarly, the behavior of a CHRO who views an employment metric indicator shortly after (within a threshold period) the headcount metric indicator can also cause the computer system to identify the relationship between the employment metric indicator and the headcount metric indicator, and in a graph of relationships where the strength of that relationship is defined by the weight on the edge and the nodes are metrics, cause a weight to be placed on that edge. A metric indicator can have associated metadata that can be used by the computer system to create a relationship between two metric indicators. Optionally, a metric indicator can have any number of relationships. In the above example, the headcount metric indicator has two associated metric indicators, and the headcount decrease metric indicator and the employment metric indicator each have one. Optionally, the computer system may identify relationships by analyzing user logs and finding indicators that are viewed close in time based on the timestamps in the user logs. In some embodiments, the user logs of multiple users can be used to generate relationships between metric indicators used for all other users.

[0014] The behavior of metric indicators over time can also be used to identify relationships between metric indicators. For example, over time, a computer system can analyze the values (trends) of metric indicators. The computer system analysis method can, for example, identify leading indicators and lagging indicators. One example is that when inventory begins to trend downward, revenue also trends downward, but in some cases, later (e.g., 3 days later, 2 weeks later, 1 month later, etc.). The computer system can analyze the similarity of trends with a time delay for the revenue trend, identify the inventory metric indicator as a leading indicator for the revenue metric indicator, and conversely, identify the revenue metric indicator as a lagging indicator for the inventory metric indicator. The metadata associated with each metric indicator can be used to store the relationship and type of relationship (leading indicator / lagging indicator) between two metric indicators.

[0015] In some embodiments, a user can input the relationship between metric indicators via a user interface. Since metric indicators can be created by a user, the input of relationships within the metric indicator information can be useful for other users in creating relationships.

[0016] Relationships created using user logs, explicit user input of relationships via the user interface, and / or metric indicator behavior are used by the computer system to generate recommendations for the user based on, for example, the user's role (e.g., human resources team member, finance team member, etc.), the user's behavior, and / or the user's current view, as shown in more detail below.

[0017] One embodiment includes techniques for crowdsourcing metric indicators. In some embodiments, when the system is first set up for a business that may be provided as an on-site installation or as a cloud-based system in a software ( "SaaS": software as a service) implementation as a service, the ready-to-use dashboard for any given user can include a set of metric indicators relevant to that type of user. For example, a CEO may have different metric indicators on the dashboard than, for example, a CHRO. One of the metric indicators can be a trending metric indicator. The trending metric indicator can be provided using, for example, a tag cloud. For example, the trending metric indicator can include metric indicators being viewed by other users in the company or, in some embodiments, more specifically, by other members of the user's team. In some embodiments, when the installation is new, the trending metric indicator can be identified based on what other users in other companies are viewing as common domain knowledge within the company. The tag cloud can provide a list of metric indicator names with varying font sizes based on the popularity of the metric indicators. For example, a frequently viewed metric indicator can have a larger font than a metric indicator being viewed by only one user. In some embodiments, regardless of the period during which the business has used the system, the trending metric indicator can be configured by the user to identify the tag cloud based on the user's team, generally the enterprise, or other similar companies hosted in the same cloud-based system. The tag cloud can be generated based on tracking the user's behavior and which metric indicators are being viewed at any given time. The trending metric indicator can be updated periodically, for example, every hour or when refreshed by the user.The metric indicators listed in the trend metric indicator can be selected by the user. With the selection, the metric indicator can be added to, for example, the user's dashboard. In some embodiments, the selection of a metric indicator name from a tag cloud can provide a detailed screen that provides information associated with the selected metric indicator.

[0018] In some embodiments, the metric indicator includes metadata used to model business metrics. Each metric indicator can have several attributes. Each metric indicator definition can describe a primary value along with a detailed perspective that provides additional information about the primary value.

[0019] In some embodiments, the metric indicator can be shared across various business domains (e.g., customer experience ("CX"), human capital management ("HCM"), enterprise resource planning ("ERP"), supply chain management ("SCM")). Each metric indicator can be informed by personas and business goals within those specific domains. The user interface can provide an immediate value and can morph over time to best serve the business. Thus, the metric indicator can reflect prior knowledge of the business and "learn" over time. The user interface can morph (change) because, for example, a new metric indicator can be used as a default for the user in the user interface, and different attributes for each metric indicator may be shown. Further, proposals can be provided that enable the user to modify the details of the dashboard / user interface.

[0020] During initial installation (e.g., site installation or cloud-based implementation), two or more metric indicators can be defined and provided as part of the application. After installation, the metric indicators can be organizationally defined and continue to be picked up (selected for use) by the user. Metric indicators can also occur organically over time to surface new or changing business needs. Metric indicators can reflect previous knowledge of the business and can "become" selected over time as recommendations from the user's usage history. Metric indicators can utilize data to model the business. The data can be provided by industry benchmark data, third-party data sources, sales and production data sources, etc.

[0021] Referring now to FIG. 1, a smart analytics system 100 is shown in a high-level block diagram. Using the smart analytics system 100, metric indicators (KPIs and cards) can be generated for presentation to the user. The smart analytics system 100 includes a user device 105, a smart analytics server 110, a smart analytics database 115, and an enterprise data source 120. The smart analytics system 100 can include additional components not shown for simplicity in FIG. 1.

[0022] The user device 105 can be any suitable user computing device such as the user devices 1002, 1004, 1006, or 1008 of FIG. 10, the client devices 1104, 1106, 1108 of FIG. 11, or the computer system 1200 of FIG. 12. The user can use the user device 105 to access the user interface 125 of the smart analytics server 110 to obtain a graphical user interface showing KPI cards and sets.

[0023] The smart analysis server 110 can be a computer system such as the computer system 1200 of FIG. 12 or the server 1012 of FIG. 10. The smart analysis server 110 includes a user interface 125 and a KPI subsystem 130. The user interface 125 can be used by the user device 105 to obtain access to the information generated by the KPI subsystem 130. The KPI subsystem 130 generates metric indicators (KPIs), generates recommendations for the metric indicators, or otherwise identifies high-value content for the user to view, as described in more detail in FIG. 13. The KPI subsystem 130 accesses the smart analysis database 115 to store and retrieve the information generated by the smart analysis server 110. The smart analysis database can include a metadata index 1330, an important KPI and relationship database 1332, a role and KPI database 1334, and the action knowledge base 1336 of FIG. 13.

[0024] The enterprise data source 120 can be any data source of the enterprise that provides data used in creating KPIs. This information can be retrieved from the enterprise data source 120 by the smart analysis server 110 to perform metric indicator analysis, as described in more detail below.

[0025] FIG. 2A shows an example dashboard 200 for a CHRO. A user can log in to the dashboard / user interface, which can be hosted on a computer system such as, for example, computer system 1200 of FIG. 12 , cloud infrastructure system 1102 of FIG. 11 , or server 1012 of FIG. 10 . Dashboard 200 can include multiple metric indicator cards 215 and 220. Cards 215 and 220 can each provide a view of activity represented by metric indicators. Metric indicator cards 215 and 220 can each represent information about a particular metric. The information may be stored in a data store of information for the enterprise. Each user can group various metrics on the metric indicator cards into one or more sets. In some embodiments, the sets are created by the system or grouped based on metadata underlying each of the metric indicator cards 215 and 220. Each metric indicator card can include metadata containing information (e.g., identifiers) about other associated metric indicator cards. Metric indicator cards 215 and 220 may each be created by a user of the system, created by the system based on user behavior, or provided with the initial implementation of the system.

[0026] Metric indicator cards 215 and 220 may be generated by a user by selecting metrics to view along with metadata. These cards may be personalized by each user while still relying on the same underlying data. For example, metric indicator cards 215 and 220 may show line graph trends for one user, while another user prefers a pie chart display. These types of user-specific personalizations are stored in metadata for that user within the metric indicator cards. Additional information in the metadata of metric indicator cards 215 and 220 includes relationship data regarding the associated metric indicator. The relationship data may be generated manually, by rules, and / or using machine learning.

[0027] Dashboard 200 may include a section that provides notifications 205. Notifications 205 may include team notifications applicable to all users in a team. Notifications 205 may provide information about specific users (e.g., Linda and Thomas) that is useful to all members of the team. Alerts 210 may provide relevant suggestions and notifications to the user. For example, alert box 210 includes an alert that new job attrition is at 12%. New job labor turnover rate metric indicator 215 shows the user further data related to this alert. A second alert 210 also states that the top quartile of employees in U.S.-based manufacturing attrition is 4.8 times higher than the expected rate, which may be useful information to a user in a CHRO user interface. Alerts are clickable so that selecting the alert can provide a view of metric indicators that provide relevant data about the attrition information.

[0028] If an alert is selected, the user interface 250 of FIG. 2B may be provided to the user. The user interface 250 may include information relevant to the alert. For example, the user interface 250 may include further insights 255 into data relevant to high attrition rates. The leading indicator 260 may provide the user with further insights regarding high levels of attrition and employee departures. The high risk metric indicator 265 may identify parts of the company that are at risk for additional attrition based on historical data. The loss metric indicator 270 may provide information about employees who left in the most recent quarter. The attrition metric indicator 275 may provide monthly attrition information based on region (e.g., Europe and Africa). The attrition by department metric indicator 280 may provide quarterly information on attrition organized by department.

[0029] Dashboard 250 may be one view of a drill-down of metric indicators based on a selected metric indicator from initial dashboard 200. In some embodiments, different metric indicators may be configured to be displayed based on the selected metric indicator.

[0030] 3 may be another example dashboard 300. For example, a finance team member may be viewing income statement metric indicators, including income statement graph 305. A user may select a revenue bar in income statement graph 305, which may display revenue metric indicator 310. Because of relationships identified between revenue metric indicator 310 and direct revenue metric indicators, indirect revenue metric indicators, sales velocity metric indicators, revenue-to-Amazon metric indicators, sales volume metric indicators, etc., metric indicators in associated smart indicator selection box 315 may be displayed for user selection and viewing.

[0031] In some embodiments, the metric indicators shown in the associated smart indicator selection box 315 can be selected based on relationships created based on the usage behavior of one or more users (e.g., analysis of the user logs of the users), explicit relationship creation by the user via the user interface, and / or automated relationship creation based on a computer system that analyzes the metric indicator behavior over time. In some embodiments, the behavior of the viewing user can be utilized to select the metric indicators shown in the associated smart indicator selection box 315. For example, if a user has recently viewed a certain metric indicator, it can be excluded (or included) in the associated smart indicator selection box 315.

[0032] FIG. 4 can be a flowchart 400 of a method for providing data-driven correlations of metrics. The method of flowchart 400 can be executed by a computer such as, for example, server 1012 of FIG. 10, cloud infrastructure system 1102 of FIG. 11, or computer system 1200 of FIG. 12. Flowchart 400 can begin with step 405 of providing a dashboard to a user via a user interface, the dashboard including a visual depiction of metric indicators that the dashboard is configurable for each user. For example, user interfaces 200 or 250 provide metric indicators for human resource (「HR」) users. Different types of users may want to view financial information (e.g., revenue metric indicators), and accordingly, can configure the dashboard. In some embodiments, an initial dashboard configuration can be provided to various types of users. For example, an HR user can select an HR dashboard configured with headcount metric indicators, hiring metric indicators, headcount reduction metric indicators, etc. As another example, a finance user may select a finance dashboard configured with revenue metric indicators, sales metric indicators, etc. Each user can then remove and add metric indicators to make the dashboard unique to that user. The method can continue in step 410 by tracking and continuing the user's usage behavior in the user interface. For example, an HR user can typically view a metric indicator for the headcount reduction rate for the most recent quarter in the United States when the user logs in. This can be identified as a usage pattern. Each of the user activities may be stored in a user log. The user log can include, for example, a username, activity, and timestamp. The method can follow step 415 of identifying a relationship between a first metric indicator and a second metric indicator.For example, the user log information may indicate that the user views the first metric indicator and the second metric indicator within each other's threshold periods. The method may follow step 420 of providing a proposal to view the second metric indicator when a first user is viewing the first metric indicator on the dashboard. For example, the user interface 300 provides an exemplary view with a proposal. The associated smart metric indicator selection box 315 provides a proposal for an associated metric indicator based on the user viewing the revenue metric indicator 310.

[0033] Returning to step 415, identifying the relationship between the first metric indicator and the second metric indicator may include identifying the relationship because the user manually entered the relationship via the user interface, for example. The user may be able to tag and mark metric indicators as related. For example, the user can select two metric indicators, mark the first metric indicator as leading (i.e., the trend of this metric indicator precedes that of the second metric indicator), and mark the second metric indicator as lagging (i.e., the trend of this metric indicator follows the trend of the second metric indicator). The metadata stored with the metric indicator enables this tagging and relationship creation. The metadata may be stored in the data repository 1014, for example, as described with respect to FIG. 10.

[0034] In step 415, identifying the relationship between the first metric indicator and the second metric indicator may include identifying the relationship based on a user log that records user behavior and activities in the user interface. For example, a sample usage log is shown in Table 1.

[0035]

Table 1

[0036] Using the logs shown in Table 1 and using data mining based on association rule learning, frequently co-viewed metric indicators are identified and an association is made between those metric indicators. This analysis may identify that there is a likelihood of a relationship without identifying the type of relationship. For example, using the example log in Table 1, "gross profit" and "regional revenue" can be identified as having a relationship because multiple users execute them sequentially during a session. The association rule may identify that the metric indicators were executed in the same session by the same user or were executed by the same user within a specific period. The sequence of execution may provide insights in the association rule algorithm. The association may include additional information including the type of user for which the relationship applies (user role and / or area). Using the above example, the area of the user may be the CFO and the role of the user may be a financial officer. User logs may be collected for each user within the enterprise.

[0037] In step 415, identifying the relationship between the first metric indicator and the second metric indicator may include automatically identifying the relationship by analyzing the trends of the metric indicators. Using the trends of the metric indicator values, associated metric indicators and the type of relationship (e.g., leading / lagging metric indicators) may be identified. Periodically (e.g., daily, weekly, monthly, quarterly, annually, etc.), snapshots of the values of each metric indicator can be created. The snapshots can be used to identify any trends, patterns, and / or interdependencies. For example, if the first metric indicator increases and then the second metric indicator also increases, and this trend is consistently shown, a relationship can be inferred between the two metric indicators. The period used to identify whether two metric indicators share trends or one leads / follows the other can vary. For example, some metric indicators share trends and can be seen in each snapshot, while in other relationships, the second metric indicator can lag another metric indicator by several months. As another example, if the third metric indicator increases and then the fourth metric indicator decreases, a relationship can be inferred between the metric indicators. Table 2 provides exemplary metric indicator data snapshot values.

[0038]

Table 2

[0039] Table 2 provides a snapshot of values collected and stored over a two-day period for three metric indicators (gross profit, revenue, expenses). Such metric indicator snapshots can be collected across all hosted customers in a cloud-based hosted environment. Algorithms can be applied to the snapshot data to identify trends in the metric indicator values. Correlations between metric indicator values that consistently trend together (e.g., over time, both metric indicators increase and / or decrease together, or one metric indicator increases and then after some time the other metric indicator increases and / or decreases) are used to identify that the metric indicators are related and the type of relationship (e.g., leading / lagging).

[0040] Figure 5 shows another exemplary user interface 500 that can be, for example, an initial out-of-the-box dashboard provided to a user. When a user utilizes the user interface, proposals can be provided based on the user's usage patterns and current data associated with those usage patterns using the tracking described above.

[0041] The user interface 500 can be, for example, a dashboard for the CEO. The CEO user interface 500 can include metric indicators such as quarterly revenue 505 and year-to-date (YTD) revenue 510. The CEO user interface 500 can also include trend metric indicators 515. The trend metric indicator 515 can be a tag cloud that provides trend metric indicator information (also described herein as cloud-source metric indicator information). More popular metric indicators are shown in a larger font. In the user interface 500, for example, the employee diversity KPI is the most popular metric indicator, as currently displayed and indicated by the largest font. In some embodiments, color coding may be further used to emphasize the most popular / currently trending metric indicators. For example, a bright color may be used for the most trending ones, while a dimmer color, gray, or other less prominent color coding may be used for less popular metric indicators at the time when trend information is collected. As the user browses the user interface 500, the trend metric indicator 515 may change as the trend of the metric indicator changes. For example, as the profit KPI becomes more popular (more frequently used by others in the company, team, etc.), the font size of "profit" within the trend metric indicator 515 may increase. Similarly, for example, as the popularity of the employment rate KPI decreases (fewer users viewing it in the company, team, etc.), the font size of "employment rate" within the trend metric indicator 515 may decrease. In some embodiments, the user may select a metric indicator within the trend metric indicator 515, for example, by clicking on the metric indicator name within the trend metric indicator 515.Selecting a metric indicator can drill down into the metric indicator, and / or display a more detailed screen with the metric indicator and other underlying information about the metric indicator, to select the metric indicator. In some embodiments, the user can select whether to analyze the trend of the metric indicator based on, for example, the trend metric indicator 515 being for all users within the enterprise, users belonging to a specific team within the enterprise, users belonging to a specific team across other enterprises if information about other enterprises is available, etc.

[0042] FIG. 6 shows a flowchart 600 illustrating a method for crowdsourcing metric indicators. The method may be performed by, for example, the server 1012 of FIG. 10 , the cloud infrastructure system 1102 of FIG. 11 , or the computer system 1200 of FIG. 12 . The method may include, at step 605, providing a plurality of metric indicators in a user interface. In some embodiments, the metric indicators are all generated by a software manufacturer. In some embodiments, the metric indicators may be generated and / or customized by a user. The method may further include, at step 610, tracking a plurality of users using the user interface. In some embodiments, the users may be teammates of the individual using the user interface. In some embodiments, the users may be other users within the individual using the user interface's company. In some embodiments, the users may be from other companies (enterprises). The method may further include, at step 615, identifying trending data for each of the metric indicators based on tracking user usage. For example, if a large number of users (e.g., 15 users in a 40-user company) are viewing a metric indicator, the metric indicator may be considered trending. Determining whether a metric indicator is trending upward or downward may include analyzing whether the number of users viewing the metric indicator is increasing or decreasing over a period of time (e.g., 30 minutes, 24 months, 3 quarters, 13 weeks). Additionally, the number of users viewing a first metric indicator can be considered in relation to the number of users viewing other metric indicators to determine whether the first metric indicator is popular.Additionally, the total number of users in the system may further indicate, for example, the popularity of a metric indicator based, for example, on the percentage of users viewing that metric indicator (e.g., 90% of users viewing the metric indicator may indicate high popularity and a large font in the tag cloud, while 2% of users viewing the metric indicator may indicate low popularity and no metric indicator in the tag cloud of trending metric indicators 515). Metric indicators with low attention and / or views may have low trend values. Metric indicators with declining views may also have low trend values. The method may also include, in step 620, providing trend data to one or more users in a user interface with suggestions and recommendations for viewing these KPIs. For example, a tag cloud identifying trending metric indicators, such as trending metric indicator 515 of FIG. 5, may be generated. The tag cloud may provide a clickable list showing metric indicator names and providing information about the trend value or popularity of the named metric indicator. For example, the most popular metric indicators may be displayed in a large font in the clickable list. Optionally, the method may include receiving a selection of a metric indicator named in the clickable list, and in response to the selection, providing a user via a user interface a view of details of the selected metric indicator.

[0043] Returning to Figure 5, user interface 500 may include an option to add a metric indicator 520. Selecting add metric indicator 520 may provide the user interface shown in user interface 700 of Figure 4. User interface 700 may provide a screen 705 that allows the user to select the type of selected metric indicator.

[0044] FIG. 8 may show a user interface 800 that may be used to enter data to be associated with a metric indicator. The base, interaction, and relationship, when entered by a user, provide metadata to be associated with the metric indicator. For example, a target persona may identify the type of user to target. As another example, associated metric indicators may be identified to cross-reference options (to create relationships for recommendations, as described with respect to FIGS. 3 and 4). Each of the metadata values may further be used to identify relevant results when a user searches within the user interface.

[0045] Within the user interface, "new" metric indicators can be data-driven based on the needs of the consuming persona and can reflect any number of Oracle and non-Oracle data sources. New metric indicators can be configurable by the user and can be viewed across multiple touchpoints based on the user's preferences. New metric indicators can be pushed (alerts) to the user based on configurable thresholds, industry benchmarks, or business urgency derived from advanced intelligence ("AI")-driven monitoring. Metric indicators can further serve as a focal point for applications. Metric indicators can also be configurable by the user. Metric indicators can be viewed across multiple touchpoints based on the user. Metric indicators can filter all available metric indicators based on a range of parameters, not just those currently shown on their user interface. Search can provide a mechanism for simple and complex searching / filtering. Easy controls can be provided for filtering and surfacing sets of metric indicators.

[0046] In some embodiments, the search parameters can include, for example, subject / business goals, KPI status and status severity, usage / trends, seasonality, etc. In some embodiments, the search can always be present on the user interface. The search can be integrated with the Ask function. Further, the search may not exist based on the derived relationships.

[0047] In some embodiments, the system can utilize metadata tags and relationships derived from application usage as described with respect to FIGS. 3 and 4.

[0048] The application / user interface can be driven by a consistent definition of metric indicators that can be surfaced as desired by the user. An example of configuring the metric indicator can include setting personalized thresholds that override the default of the metric indicator. The types of thresholds can include personal thresholds, benchmarks, and automatic / AI detection. The metric indicator can further include custom navigate, multiple thresholds, override directionality, custom navigate to subsequent processes, general rules, override rules, and / or custom tags. Based on the user's preferences regarding the thresholds defined in the user interface and the mechanism for updating with respect to the metric indicator, the user interface and / or the server system can provide those updates. The user can have the ability to set the manner of sending notifications of different severities, such as text, email, voice, etc. for a mobile device. Where possible, it is also possible to provide a link back to the original metric indicator and a next step action along with the notification.

[0049] Within the user interface, the metric indicator can provide a snapshot into the business and a starting point for a number of subsequent actions. These next steps are intended to further describe or explore the business and take action on issues or opportunities revealed by the metric indicator. The next steps range from tightly defined automated drill-downs to fully customizable navigation paths. Ultimately, the metric indicator can effectively serve as a gateway to rich capabilities.

[0050] In some embodiments, drill-down to details may be available. Figure 9 is an exemplary metric indicator detail view 900 of a monthly revenue metric indicator. For example, based on metadata including derived data and metric indicator details, a selection of related indicators can be selected for the drill-down menu of the monthly revenue metric indicator. The drill-down analysis is automatically generated and mapped to a predefined pattern. An optimal pattern can be automatically selected based on the metric indicator type (e.g., focused on time, leading indicators, aggregate information requiring breakdown, etc.) and a set of metric indicator attributes. This selection can also be overridden at the metric indicator definition level. Even with the default selection, other drill-down patterns are still indicative and the user can still view them if they select them.

[0051] Using observations of multiple attributes of the metric indicator itself and how they are being used, it is possible to drive the automated generation of an insightful analytical experience, also called a drill-down screen. This approach eliminates the burden of manually creating content from the user and reveals insights that would otherwise be missed. First, the definition of metric indicator metadata and metric indicator types can directly determine and populate the drill-down detail screen. An example could be rendering the range of information permutations captured by the metric indicator to present the user with information broken down by dimension or expanded by prompt. Second, the drill-down can also be adjusted based on the user's preferences (thresholds). Personal thresholds can inform the relevance for a particular user and can also be aggregated across a user population. Data captured around user behavior and system usage provides a third input. The adjusted drill-down detail template can include specific metric indicators, usage-based related metric indicators, leading / lagging indicators, related analysis, and factors that contribute to, are associated with, or explain them. The detailed view may span multiple metric indicators or incorporate multiple metric indicators. Fourth, insights derived from machine learning algorithms (identifying patterns or anomalous events across large amounts of data) can also be collected from system data and system usage, attached to the metric indicators, and surfaced in these metric indicator drill-downs. Classic (TIME) Historical & Predictive Analysis also informs the state of the business. Since this is a dynamic system where usage, data, and content are constantly changing, the output morphs over time based on the state of the application. Related metric indicators, leading / lagging indicators, or predictive analysis are good examples.

[0052] Some embodiments can include a method for providing a metric indicator authoring approach for dynamic analysis. The method can be implemented by a computer system. The method can include providing a user interface to a user. The method can further include receiving, via the user interface, a definition of the metric indicator including metadata associated with the metric indicator. For example, the selection of the save button in FIG. 8 can send a value entered via the user interface to the computer system. The method can further include generating a metric indicator detail view including data associated with the metric indicator based on the metadata. The detail view can be similar to, for example, the detail view 900 of FIG. 9. The method can further include providing the detail view via the user interface.

[0053] The infrastructure can be realized in various different environments including cloud environments (which can be various types of clouds including private cloud environments, public cloud environments, and hybrid cloud environments), on-premises environments, hybrid environments, and the like.

[0054] FIG. 10 shows a schematic diagram of a distributed system 1000 for implementing an embodiment. In the illustrated embodiment, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to a server 1012 via one or more communication networks 1010. The client computing devices 1002, 1004, 1006, and 1008 can be configured to execute one or more applications.

[0055] In certain embodiments, server 1012 may provide services or software applications that may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as a software as a service (SaaS) model, to users of client computing devices 1002, 1004, 1006, and / or 1008. Users operating client computing devices 1002, 1004, 1006, and / or 1008 may utilize one or more client applications to interact with server 1012 to utilize the services provided by these components.

[0056] In the configuration shown in FIG. 10, server 1012 may include one or more components 1018, 1020, and 1022 that implement functions executed by server 1012. These components may include software components that may be executed by one or more processors, hardware components, or combinations thereof. It should be recognized that a wide variety of system configurations may be possible that may differ from distributed system 1000. Accordingly, the embodiment shown in FIG. 10 is an example of a distributed system for implementing an embodiment of the system and is not intended to be limiting.

[0057] A user may use client computing devices 1002, 1004, 1006, and / or 1008 to execute one or more applications, which may generate one or more memory requests in accordance with the teachings of the present disclosure and then process them. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via this interface. Although FIG. 10 shows only four client computing devices, any number of client computing devices may be supported.

[0058] The client device can include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, game systems, thin clients, various messaging devices, sensors or other sensing devices. These computing devices can include various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-based operating systems, Linux® or Linux-based operating systems, such as various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, Palm OS®), Google Chrome® OS). Portable handheld devices can include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices can include Google Glass® head-mounted displays and other devices. Game systems can include various handheld game devices, Internet-connected game devices (e.g., Microsoft Xbox® game consoles with / without Kinect® gesture input devices, Sony PlayStation® systems, various game systems provided by Nintendo®, etc.).The client device may be capable of running a wide variety of applications, such as various internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use a variety of communication protocols.

[0059] Network 1010 may be any type of network known to those skilled in the art that is capable of supporting data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (transmission control protocol / Internet protocol), SNA (systems network architecture), IPX (Internet packet exchange), AppleTalk®, etc. By way of example only, network 1010 may include a local area network (LAN), an Ethernet-based network, a token ring, a wide-area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a wireless network operating under any of the IEEE (Institute of Electrical and Electronics) 802.11 protocol suite, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0060] Server 1012 may be composed of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, UNIX (registered trademark) servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable configurations and / or combinations. Server 1012 may include one or more virtual machines that execute a virtual operating system, or other computing architectures with virtualization. This may be, for example, one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for the server. In various embodiments, Server 1012 may be adapted to execute one or more services or software applications that provide the functions described in the above disclosure.

[0061] The computing system within Server 1012 may include one or more operating systems including any of the above operating systems, and may execute commercially available server operating systems. Also, Server 1012 may execute any of various other server applications and / or middle-tier applications including, for example, an HTTP (hypertext transport protocol) server, an FTP (file transfer protocol) server, a CGI (common gateway interface) server, a JAVA (registered trademark) server, a database server, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle (registered trademark), Microsoft (registered trademark), Sybase (registered trademark), IBM (International Business Machines), etc.

[0062] In some implementations, server 1012 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client computing devices 1002, 1004, 1006, and 1008. As an example, the data feeds and / or event updates may include real-time events related to sensor data applications, financial stock market dashboards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, etc., and may include, but are not limited to, Twitter (registered trademark) feeds, Facebook (registered trademark) updates, or real-time updates received from one or more third-party information sources and continuous data streams. Server 1012 may also include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client computing devices 1002, 1004, 1006, and 1008.

[0063] The distributed system 1000 may also include one or more data repositories 1014, 1016. In certain embodiments, these data repositories can be used to store data and other information. For example, one or more of the data repositories 1014, 1016 can be used to store information such as data used for analysis and display on dashboards. The data repositories 1014, 1016 can be located in various places. For example, the data repository used by the server 1012 can be at the local location of the server 1012 or at a remote location from the server 1012 and communicate with the server 1012 via a network-based connection or a dedicated connection. The data repositories 1014, 1016 can be of different types. In certain embodiments, the data repository used by the server 1012 can be a database, such as a relational database provided by Oracle Corporation (registered trademark) and other manufacturers. One or more of these databases can be adapted to enable storage, update, and retrieval of data in response to SQL-format commands.

[0064] In certain embodiments, one or more of the data repositories 1014, 1016 may be used by an application to store application data. The data repository used by the application can be of various types, such as, for example, a key-value store repository, an object store repository, or a general-purpose storage repository supported by a file system.

[0065] In particular embodiments, the storage-related functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 11 is a simplified block diagram of a cloud-based system environment 1100 that may provide various storage-related services as cloud services, according to particular embodiments. In the embodiment shown in FIG. 11, a cloud infrastructure system 1102 may provide one or more cloud services that users may request using one or more client computing devices 1104, 1106, and 1108. The cloud infrastructure system 1102 may include one or more computers and / or servers, which may include those described above with respect to server 1012. The computers in the cloud infrastructure system 1102 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0066] The network 1110 may facilitate communication and exchange of data between the clients 1104, 1106, and 1108 and the cloud infrastructure system 1102. The network 1110 may include one or more networks. The networks may be of the same type or different types. The network 1110 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0067] The embodiment shown in FIG. 11 is merely an example of a cloud infrastructure system and is not intended to be limiting. In some other embodiments, it should be understood that the cloud infrastructure system 1102 may have more or fewer components than those shown in FIG. 11, may combine two or more components, or may have components with different configurations or arrangements. For example, although FIG. 11 shows three client computing devices, in alternative embodiments, any number of client computing devices may be supported.

[0068] The term cloud service is generally used to refer to services made available to users on demand via a communications network, such as the Internet, by a service provider's system (e.g., cloud infrastructure system 1102). Typically, in a public cloud environment, the servers and systems that comprise the cloud service provider's system are distinct from a customer's own on-premise servers and systems. The cloud service provider's systems are managed by the cloud service provider. Thus, customers may use cloud services offered by a cloud service provider without purchasing separate licenses, support, or hardware and software resources for the services. For example, a cloud service provider's system may host applications, and users may order and use the applications on demand via the Internet without purchasing infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, and Java cloud services, are offered by Oracle Corporation of Redwood Shores, California.

[0069] In certain embodiments, the cloud infrastructure system 1102 can provide one or more cloud services using various models, including a software as a service (SaaS) model, a platform as a service (PaaS) model, an infrastructure as a service (IaaS) model, and a hybrid service model. The cloud infrastructure system 1102 can include a suite of applications, middleware, databases, and other resources that enable the delivery of various cloud services.

[0070] The SaaS model enables applications or software to be delivered to customers as a service over a communication network such as the Internet without the customer having to purchase the underlying hardware or software for the applications. For example, by using the SaaS model, customers can be given access to on-demand applications hosted by the cloud infrastructure system 1102. Examples of SaaS services provided by Oracle Corporation (registered trademark) include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and the like.

[0071] The IaaS model is generally used to provide flexible computing and storage capabilities by providing infrastructure resources (e.g., servers, storage, hardware, and networking resources) to customers as cloud services. Various IaaS services are provided by Oracle Corporation (registered trademark).

[0072] The PaaS model is generally used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without having to procure, build, or manage the environment resources. Examples of PaaS services provided by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0073] Cloud services are typically provided on an on-demand, self-service basis, on a subscription basis, and in a flexible, scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services offered by cloud infrastructure system 1102 via a subscription order. Cloud infrastructure system 1102 then performs processing to provide the services requested in the customer's subscription order. Cloud infrastructure system 1102 may also be configured to provide one cloud service or multiple cloud services.

[0074] The cloud infrastructure system 1102 can provide cloud services via various deployment models. In the public cloud model, the cloud infrastructure system 1102 may be owned by a third-party cloud service provider, and the cloud services are provided to general public customers. This customer can be an individual or an enterprise. In certain other embodiments, under the private cloud model, the cloud infrastructure system 1102 may function within an organization (e.g., within a corporate organization), and the services are provided to customers within this organization. For example, this customer may be various departments of an enterprise such as the human resources department, the payroll department, or an individual within the enterprise. In certain other embodiments, under the community cloud model, the cloud infrastructure system 1102 and the provided services may be shared among various organizations within the relevant community. Other various models such as hybrid models of the above models may also be used.

[0075] The client computing devices 1104, 1106, and 1108 may be of different types (e.g., devices 1002, 1004, 1006, and 1008 shown in FIG. 10), and may be operable with one or more client applications. A user can interact with the cloud infrastructure system 1102, such as by using the client device to request services provided by the cloud infrastructure system 1102.

[0076] In some embodiments, the processing performed by cloud infrastructure system 1102 may include big data analytics. This analytics may involve using large data sets, analyzing, and processing them to detect and visualize various trends, behaviors, relationships, etc. within this data. This analytics may be performed by one or more processors, possibly processing the data in parallel, running simulations with the data, etc. The data used in this analytics may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0077] 11 , cloud infrastructure system 1102 may include infrastructure resources 1130 utilized to facilitate the provision of various cloud services offered by cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, etc.

[0078] In certain embodiments, to facilitate the efficient provisioning of these resources to support the various cloud services provided by the cloud infrastructure system 1102 to different customers, the resources may be grouped into sets of resources (also referred to as "pods") or resource modules. Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In certain embodiments, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, and a second set of pods, which may include a different combination of resources than the pods within the first set of pods, may be provisioned for Java services or the like. For some services, the resources allocated to provision these services may be shared among the services.

[0079] The cloud infrastructure system 1102 itself may internally use a service 1132 that is shared by different components of the cloud infrastructure system 1102 and that facilitates the provisioning of services by the cloud infrastructure system 1102. These internal shared services may include, but are not limited to, security identity services, integration services, enterprise repository services, enterprise manager services, virus scan whitelist services, high availability, backup recovery services, services enabling cloud support, email services, notification services, file transfer services, and the like.

[0080] The cloud infrastructure system 1102 may include a plurality of subsystems. These subsystems may be implemented in software, or in hardware, or a combination thereof. As shown in FIG. 11, the subsystem may include a user interface subsystem 1112 that enables a user or customer of the cloud infrastructure system 1102 to interact with the cloud infrastructure system 1102. The user interface subsystem 1112 may include various different interfaces such as a web interface 1114, an online store interface 1116 where cloud services provided by the cloud infrastructure system 1102 are advertised and can be purchased by consumers, and other interfaces 1118. For example, a customer may use a client device to request (service request 1134) one or more services provided by the cloud infrastructure system 1102 using one or more of the interfaces 1114, 1116, and 1118. For example, a customer may access an online store, browse the cloud services provided by the cloud infrastructure system 1102, and place a subscription order for one or more services that the cloud infrastructure system 1102 provides and the customer desires to subscribe to. This service request may include information identifying the customer and one or more services that the customer desires to subscribe to. For example, a customer may place an order to subscribe to a storage-related service provided by the cloud infrastructure system 1102. As part of the order, the customer may provide information identifying the application for which the service is to be provided and application storage profile information about that application.

[0081] In certain embodiments, such as the embodiment shown in FIG. 11, the cloud infrastructure system 1102 may include an order management subsystem (OMS) 1120 configured to process new orders. As part of this process, the OMS 1120 creates a customer account if one has not already been created, receives billing and / or account information from the customer for use in billing the customer for the requested services provided to the customer, verifies the customer information, and upon verification, reserves the order for the customer and may be configured to prepare the order for provisioning by coordinating various workflows.

[0082] Once properly authenticated, the OMS 1120 may call an order provisioning subsystem (OPS) 1124 configured to provision resources for this order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring the resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for an order and the type of resources provisioned may depend on the type of cloud service the customer has ordered. For example, according to one workflow, the OPS 1124 may be configured to determine the specific cloud service requested and identify the number of pods that would have been pre-configured for this specific cloud service. The number of pods allocated for an order may depend on the size / volume / level / range of the service requested. For example, the number of pods to allocate may be determined based on the number of users the service is to support, the period for which the service is requested, etc. Next, the allocated pods may be customized to the specific customer making the request to provide the requested services.

[0083] The cloud infrastructure system 1102 may send a response or notification 1144 to the requesting customer to indicate when the requested service will be available. In some examples, the customer may be sent information (e.g., a link) that enables the customer to begin using and utilizing the benefits of the requested service.

[0084] The cloud infrastructure system 1102 may provide services to multiple customers. For each customer, the cloud infrastructure system 1102 manages information related to one or more subscription orders received from the customer, maintains customer data related to the orders, and serves to provide the requested services to the customer. Also, the cloud infrastructure system 1102 may collect usage statistics regarding the use of the subscribed services by the customers. For example, the statistics may be collected regarding the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information may be used to bill the customers. Billing may be done, for example, monthly.

[0085] The cloud infrastructure system 1102 may provide services to multiple customers in parallel. The cloud infrastructure system 1102 may store information about these customers, which may include copyright information in some cases. In certain embodiments, the cloud infrastructure system 1102 includes an identity management subsystem (IMS) 1128 configured to manage customer information and separate the information being managed so that information about one customer is not accessible from information about another customer. The IMS 1128 may be configured to provide various security-related services, such as services for managing identity services, information access management, authentication and authorization services, customer identities and roles, and related capabilities.

[0086] 12 illustrates an exemplary computer system 1200 that may be used to implement certain embodiments. As shown in FIG. 12, computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with several other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. Storage subsystem 1218 may include non-transitory computer-readable storage media, including storage medium 1222 and system memory 710.

[0087] Bus subsystem 1202 provides a mechanism for allowing the various components and subsystems of computer system 1200 to communicate with each other as intended. While bus subsystem 1202 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1202 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus, etc., using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0088] The processing subsystem 1204 controls the operation of the computer system 1200 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processor may include a single-core or multi-core processor. The processing resources of the computer system 1200 can be organized into one or more processing units 1232, 1234, etc. The processing unit may include one or more processors, one or more cores from the same or different processors, a combination of a core and a processor, or some other combination of a core and a processor. In some embodiments, the processing subsystem 1204 may include one or more dedicated coprocessors such as a graphics processor, digital signal processors (DSPs), etc. In some embodiments, some or all of the processing units of the processing subsystem 1204 may use customized circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0089] In some embodiments, the processing unit within processing subsystem 1204 may execute instructions stored in system memory 1210 or computer-readable storage medium 1222. In various embodiments, the processing unit may execute various programs or code instructions and may maintain multiple programs or processes that are executed simultaneously. At any given point in time, some or all of the program code to be executed may reside in system memory 1210 and / or computer-readable storage medium 1222, which may potentially include one or more storage devices. Through appropriate programming, processing subsystem 1204 may provide the various functions described above. In an example where computer system 1200 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0090] In certain embodiments, a processing acceleration unit 1206 may optionally be provided to execute customized processing to accelerate the overall processing performed by computer system 1200, or to offload a portion of the processing performed by processing subsystem 1204.

[0091] The I / O subsystem 1208 can include devices and mechanisms for inputting information into the computer system 1200 and / or for outputting information from, or via, the computer system 1200. Generally, the use of the term "input device" is intended to include all conceivable types of devices and mechanisms for inputting information into the computer system 1200. User interface input devices can include, for example, a keyboard, a mouse or other pointing device such as a trackball, a touchpad or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices can also include motion sensing and / or gesture recognition devices such as a Microsoft Kinect (registered trademark) motion sensor, a Microsoft Xbox (registered trademark) 360 game controller, a device that provides an interface for receiving input using gestures and voice commands, which enable a user to control and interact with the input device. User interface input devices can also include gesture recognition devices such as a Google Glass (registered trademark) blink detector that detects eye movements (e.g., "blinks" while taking a photo and / or while making a menu selection) from a user and converts the eye gesture into an input to the input device (e.g., Google Glass (registered trademark)). Additionally, user interface input devices can include voice recognition sensing devices that enable a user to interact with a voice recognition system (e.g., a Siri (registered trademark) navigator) via voice commands.

[0092] Other examples of user interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser range finders, and eye-tracking devices. Also, the user interface input device may include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasonic examination devices. The user interface input device may also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, etc.

[0093] In general, the use of the term output device is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1200 to the user or another computer. The user interface output device may include a non-visual display such as a display subsystem, indicator lights, or an audio output device. The display subsystem may be a flat panel device such as one using a cathode ray tube (CRT), liquid crystal display (LCD), or plasma display, a projected device, a touch screen, etc. For example, the user interface output device may include, but is not limited to, various display devices for visually conveying text, graphics, and audio / video information such as monitors, printers, speakers, headphones, automotive navigation systems, plotters, audio output devices, and modems.

[0094] Storage subsystem 1218 provides a repository or data store for storing information and data used by computer system 1200. Storage subsystem 1218 provides a tangible non-transitory computer-readable storage medium for storing basic programming and data configurations that provide the functionality of some embodiments. Software (e.g., programs, code modules, instructions) that provides the functionality described above when executed by processing subsystem 1204 may be stored in storage subsystem 1218. The software may be executed by one or more processing units of processing subsystem 1204. Storage subsystem 1218 may also provide a repository for storing data used in accordance with the teachings of the present disclosure.

[0095] The storage subsystem 1218 may include one or more non - transient memory devices including volatile and non - volatile memory devices. As shown in FIG. 12, the storage subsystem 1218 includes a system memory 1210 and a computer - readable storage medium 1222. The system memory 1210 may include several memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution, and a non - volatile read - only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) including basic routines that help transfer information between elements within the computer system 1200, such as during startup, may typically be stored in the ROM. Typically, the RAM includes data and / or program modules that are currently being operated on and executed by the processing subsystem 1204. In some embodiments, the system memory 1210 may include multiple different types of memories such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0096] As an example, without limitation, as shown in FIG. 12, the system memory 1210 may load an application program 1212, program data 1214, and an operating system 1216 that are in execution and that may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), and the like. As an example, the operating system 1216 may include Microsoft Windows (registered trademark), Apple Macintosh (registered trademark) and / or Linux operating systems, various commercially available UNIX (registered trademark) or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google Chrome (registered trademark) OS, etc.), and / or various versions of mobile operating systems such as iOS (registered trademark), Windows (registered trademark) Phone, Android (registered trademark) OS, BlackBerry (registered trademark) OS, Palm (registered trademark) OS operating systems, and the like.

[0097] The computer-readable storage medium 1222 may store programming and data constructs that provide functionality of some embodiments. The computer-readable storage medium 1222 may provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1200. Software (programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above may be stored in the storage subsystem 1218. By way of example, the computer-readable storage medium 1222 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD-ROM, a DVD, an optical disk drive such as a Blu-Ray® disk, or other optical media. The computer-readable storage medium 1222 may include, but is not limited to, a Zip® drive, a flash memory card, a universal serial bus (USB) flash drive, a secure digital (SD) card, a DVD disk, a digital video tape, etc. The computer-readable storage medium 1222 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROM, etc., SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0098] In particular embodiments, storage subsystem 1218 may also include a computer-readable storage medium reader 1220 that may be further connected to a computer-readable storage medium 1222. Reader 1220 may be configured to receive and read data from a memory device such as a disk, flash drive, or the like.

[0099] In particular embodiments, computer system 1200 may support virtualization techniques, including, but not limited to, virtualization of processing and memory resources. For example, computer system 1200 may provide support for running one or more virtual machines. In particular embodiments, computer system 1200 may execute a program such as a hypervisor that facilitates configuration and management of virtual machines. Each virtual machine may be assigned memory, computing (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs independently from other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating systems run by other virtual machines executed by computer system 1200. Thus, potentially multiple operating systems may be running simultaneously by computer system 1200.

[0100] The communications subsystem 1224 provides an interface to other computer systems and networks. The communications subsystem 1224 serves as an interface for sending and receiving data between other systems and the computer system 1200. For example, the communications subsystem 1224 may enable the computer system 1200 to establish a communications channel over the Internet to one or more client devices to send and receive information to and from the one or more client devices.

[0101] The communications subsystem 1224 may support both wired and / or wireless communications protocols. For example, in an embodiment, the communications subsystem 1224 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), WiFi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some embodiments, the communications subsystem 1224 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0102] The communications subsystem 1224 may receive and transmit data in various formats. For example, in some embodiments, the communications subsystem 1224 may receive incoming communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., among other formats. For example, the communications subsystem 1224 may be configured to receive (or transmit) data feeds 1226 in real time from users of social media networks and / or other communications services, such as web feeds such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0103] In certain embodiments, communication subsystem 1224 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1228 and / or event updates 1230 of real-time events that are inherently continuous or infinite and have no explicit end. Examples of applications that generate continuous data include, for example, sensor data applications, financial stock market dashboards, network performance measurement tools (such as network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and the like.

[0104] Communication subsystem 1224 may be configured to communicate data from computer system 1200 to other computer systems or networks. This data may be communicated to one or more databases that can communicate with one or more streaming data source computers coupled to computer system 1200 in various different forms such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, and the like.

[0105] Computer system 1200 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® cellular phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or other data processing system. Due to the constantly changing nature of computers and networks, the description of computer system 1200 shown in FIG. 12 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in FIG. 12. Those skilled in the art will recognize other ways and / or methods for implementing various embodiments based on the disclosure and teachings herein.

[0106] FIG. 13 illustrates an exemplary data flow through an architecture 1300 and system for creating metric indicator recommendations based on identifying and selecting the most informational content for a user. The most informational content may be information about metric indicators that have any statistically significant changes associated with them or any statistically significant anomalies associated with them over a period of time. For example, a KPI may be identified as an anomaly if its trend or trend values over a particular time period are inconsistent with the distribution of expected values based on a forecast by a multivariate time series model. Providing this information to a user may be beneficial regardless of whether the user has requested that metric indicator. Furthermore, a user may be limited to the specific metric indicators to which they have access. For any given user, relevant metric indicators to which the user has access are considered. For example, if a total revenue KPI has an anomaly signal associated with it, but the user does not have access to the total revenue KPI, the total revenue KPI will not be provided to the user. Furthermore, while a user typically views headcount KPIs and attrition KPIs, if there are no anomalies or other signals associated with those KPIs, those KPIs are not always removed from the user's view. Alternatively, user behavior may be added to the signal detection to provide additional KPIs for the user to view, so that the user can see not only KPIs that the user would self-select, but also KPIs that have information that is valuable to the user.

[0107] Architecture 1300 includes databases 1302, 1304, and 1306. Databases 1302, 1304, and 1306 can be any suitable external data sources for providing corporate data to the smart analysis system. Database 1302 is an external data source that includes data from a company, such as customer data, sales data, support data, and the like. Database 1306 is data from public and other external sources, for example, data from the Department of Labor and the like. Data warehouse 1304 is a data source directly accessed by components included in the system. External data from data sources 1302 and 1306 is used to populate data source 1304 for use by the system.

[0108] Data from the data warehouse 1304 is used in the KPI selector 1310 to identify signal anomalies. The KPI selector 1310 can identify metric indicators that may indicate an anomalous signal, for example, because they may be trending outside of a threshold. To identify the metric indicator, for a given metric indicator, the KPI modeler and forecaster 1312 can generate a multivariate dynamic dependency model. Multivariate dynamic dependency models include, but are not limited to, VARIMAX (Vector Auto Regressive Integrated Moving Average with Exogenous Variables) and multivariate variable attention temporal attention long short-term memory (MM / TM) models that use time series data for the metric indicator. A first set of predictions for values of a first variable or attribute of the metric indicator is generated based on the dynamic dependency model. Values from the model can be compared to actual values to determine whether the distribution of observed values significantly diverges from the predicted prediction by comparing them with actual values and using divergence tests to find statistical deviations. Divergence tests include, but are not limited to, the Kullback-Leibler Information Test. If the statistical deviation is significant (e.g., more than two standard deviations away from the center of gravity of the distribution), the KPI selector 1310 identifies the metric indicator as being relevant or as having some significant information that indicates that a person should look at it. As such, the KPI selector 1310 selects the KPI. If there is nothing abnormal with respect to the metric indicator, the KPI is not selected for highlighting to the user. In other words, previously forecasted values of the metric indicator may be identified and compared with the current metric indicator value. An anomaly is identified when the metric indicator value deviates statistically (e.g., by two standard deviations) from the forecasted value over multiple time points.If the metric indicator model suggests a problem or anomaly, the metric indicator may be selected for inclusion in a KPI suite for the user.

[0109] In some embodiments, selected KPIs (metric indicators) may be provided to the derivative integrator 1318 without further analysis. However, in some embodiments, the KPI explainer 1314 may generate an explanation for the anomaly based on a model. For example, a model of the metric indicator, the current metric indicator value, and the values of the contributing variables at that time may be used to identify attributes that are significant to the metric indicator at the relevant time point. Derivatives of the attributes may be calculated in the model, and the most influential ones are identified because they will have the largest derivative values based on computational techniques, including, but not limited to, locally interpretable model explanations and Shapley additive explanations. The explanations serve as locally faithful linear models that act as surrogates for the original nonlinear model. Metric indicators may also be analyzed using machine learning models to determine which metric indicators have statistically significant changes over the time period of interest. Percent changes or z-scores of metric indicators may be calculated to determine the practical significance of the changes. In some embodiments, explanations are not generated, for example, because the system is unable to converge to a poor explanation or there is insufficient data to calculate such an explanation.

[0110] Once the explanation is generated, the KPI simulator 1316 can generate recommendations for improving the signal anomaly. The recommendations can be generated by running a what-if simulation of the model by changing the identified attributes with the greatest impact to reach the most or more desirable or nearest non-anomalous value of the KPI before the anomaly occurs. For example, the what-if simulation can modify the most influential attributes until the metric indicator value in the model matches the metric indicator value of the prediction. Once the values of the attribute modifications are used, recommendations can be generated. The signal anomaly, the modeled signal anomaly, the explanation, and the recommendations can be provided to the inductive integrator 1318.

[0111] The metadata indexing and tagging engine 1308 can receive data from the data warehouse 1304 and tag or apply metadata to the data, which is then sent to the metadata index database 1330 .

[0112] When the user uses the graphical user interface shown by the browser analysis 1342, activities are generated from the creation, use, and viewing of various metric indicators. Each action is recorded by the action recorder 1326. The recorded actions are analyzed to extract activity data and context data. The activity data is stored in the activity working memory 1328, and the context data is stored in the context working memory 1324. As an example, if the user logs in and immediately views the headcount metric indicator and then the headcount reduction metric indicator, the activity could be that the KPIs are viewed in sequence over a certain length of time. The context data could be, for example, the time when the user viewed the two metrics, the device, the meeting attended, etc. The activity data and context data are combined for all related actions performed over a period of time, for example, to generate the episodic memory 1322. The episodic memory 1322 is clustered into similar episodes by the clustering engine 1320. The episode information and clusters related to the context and data are stored in the relational data store 1332, the role and KPI data store 1334, and the action knowledge data store 1336. The user-specific usage data stored in the data stores 1332, 1334, and 1336 is supplied to the inductive integrator 1318. Over time, individual user episodes can reveal patterns such as the user always logging in and immediately viewing the headcount metric and then the headcount reduction metric. Further, over time, clustered data for many users is captured and can be used to generate personalized recommendations for the user based on what many other users do and the order in which they view the KPIs. All of this information can be provided to the inductive integrator.

[0113] The induction integrator 1318 generates metric recommendations using signal anomaly information from the KPI selector 1310 and the KPI modeler and predictor 1312. In some embodiments, user-specific information is combined to generate metric recommendations. For example, if a revenue metric is identified by the KPI selector 1310, the user has access to the revenue metric, and the user typically reviews the profit metric. Even if a profit metric was not captured by the KPI selector 1310, the user may be provided with a set including the revenue metric and the profit metric. Metric recommendations and other information are provided to the presentation manager 1338. Metric recommendations may be provided to the user in the related information sidebar 1340, which may be, for example, recommendations 315 of FIG. 3. The related information sidebar 1340 may be, for example, the related metric indicators 315 shown in FIG. 3. Analytics information (used to generate the metric indicator cards and sets described above) is presented to the user via a graphical user interface. Browser analytics 1342 are presented to the user using a browser interface. Mobile analytics 1344 are presented to the user using the mobile application. The data (analytics) contained in the metric indicators is the same whether the interface is a browser or a mobile device.

[0114] In order to display metric information in a browser-based analytics and / or mobile analytics 144 user interface, a graphical representation of the KPIs must be generated. That is, it is necessary to generate KPI cards. For each KPI in the system, a model of the metric indicator (e.g., a random forest or XGBoost model) is generated using the KPI modeler and predictor 1312. Each attribute of the metric indicator is included in the model as an independent variable. Using the model, the attribute with the highest entropy change can be identified, for example, by calculating the additional explanatory value of the Shapley or local interpretable model-agnostic explanations (LIME) value. When generating a graphical representation of the metric indicator, a card, such as the card shown in the exemplary user interface of this specification, including, for example, card 505 of FIG. 5, is generated for display. The card typically includes some information associated with the attribute having the highest (maximum) entropy change identified above. In this way, the most relevant content associated with the metric indicator is provided for the user to view.

[0115] FIG. 14 shows a method 1400 that can be executed using a system having an architecture 1300 as described in FIG. 13. Method 1400 can begin at step 1405, and the system receives first data associated with activities performed by the user in a graphical user interface. For example, the action recorder 1326 can receive and record activity data of the user's interaction with the metrics using browser-based analytics 1342 and / or mobile analytics 1344.

[0116] In step 1410, activity data and context data can be extracted from the first data. For example, context data can be extracted and stored in the context working memory 1324, and activity data can be extracted and stored in the activity working memory 1328. Activity information is directly obtained from user interactions with user interface elements in the form of KPIs, cards, mouse clicks, scrolls, and typing in the context of a group, and the sequence of these actions is stored as a temporal activity graph. Similarly, for each of the above actions, its preceding action, its resulting action, the user's role and responsibilities, the user's co-activity with other users, and the context regarding the user's sharing of KPIs and email activity are also attached and stored as a graph to each node in the above-mentioned activity graph.

[0117] In step 1415, episode data is generated for all related activities performed by the user from the activity data and context data. Specifically, when the user views several related metric indicators, the viewing can be recognized as an episode and stored accordingly.

[0118] In step 1420, second data of metric indicators indicating abnormal signals is received. For example, the signal anomaly selector 1310 can receive information regarding metric indicators that reveal anomalies during analysis from the data warehouse 1304.

[0119] In step 1425, the second data is modeled by the KPI modeler and the prediction unit 1312 to identify abnormal signals as described above. For example, the modeler can generate a dynamic dependency model using the time series data of the metric indicators. A prediction of a first set of values of a first variable or attribute of the metric indicator is generated based on the dynamic dependency model. A divergence test may be used to generate a statistical deviation by comparing the value from the model with the actual value. If the statistical deviation is significant (e.g., the two standard deviations mentioned above), the KPI selector 1310 identifies that metric indicator as having some significant information indicating that it is relevant or that a person should look at it. This deviation indicates an anomaly, and thus, the signal anomaly is identified and verified.

[0120] In step 1430, an explanation of the abnormal signal based on the modeled abnormal signal may be generated. Such an explanation may include, for example, factors that led the metric indicator to deviate from the threshold trend. The explanation can be found by finding the derivative of the attributes of the model using the model of the metric indicator and the current metric indicator value, where the attributes are each represented as independent variables within the model. The attributes with the highest partial derivatives are identified as having the strongest influence when the metric indicator value is at that point. These attributes are identified to explain the anomaly using techniques such as additional explanations of SHAPley or LIME.

[0121] In step 1435, metric indicator recommendations based on the episode data and the modeled anomaly signal are generated. The recommendations may include steps that can be taken to remedy the anomaly. Specifically, the KPI path directive 1316 may generate the recommendations. A what-if simulation of the model is performed by changing the values of the identified attributes that have the most influence to bring the metric indicator value to the most desirable state. For example, if the metric indicator deviates from the forecast value, the what-if simulation can attempt to modify the attributes to obtain a metric value that corresponds to the forecast value or a value within the expected distribution.

[0122] In step 1440, a metric indicator recommendation including an explanation and recommendations for addressing the abnormal signal is generated and displayed in the graphical user interface. The metric recommendation may be displayed, for example, in a sidebar while the user is viewing the selected metric indicator. For example, the metric indicator recommended for viewing may be generated as a card with several values or information, such as a graph or values showing the metric indicator value over time, in addition to information about the most relevant attributes of the metric indicator identified by the KPI explanation portion 1314 and the KPI path indication portion 1316. The user interfaces herein provide examples of KPI cards and other information that can be generated for user viewing.

[0123] In step 1445, metric indicator recommendations, including explanations and recommendations for addressing the abnormal signal, are displayed within the graphical user interface. For example, the visual depiction may be provided within the associated information sidebar 1340 or more directly on the user's screen. As an example, the information box 210 in FIG. 2A provides some notification information that new employment attrition is at 12%, and one explanation provided is that manufacturing attrition is four times higher than the expected rate. This type of information can help the user understand why values are the way they are.

[0124] Figure 15 shows an exemplary method 1500 for providing high - value metric indicators to a user. Method 1500 can be executed, for example, by the smart analysis system 100 and / or the smart analysis architecture 1300. Method 1500 begins at step 1505, where the KPI modeler and predictor 1312 generates a model for each metric indicator, and the model has each attribute as an independent variable. The model is created for each KPI to identify which attributes are most relevant to the KPI during at least the period of interest. In this way, a KPI card can be generated for any KPI that can be presented to the user. Note that for a given user, some KPIs may not be available, and thus, in some embodiments, at step 1505, KPIs that are not accessible to the user may not be analyzed.

[0125] At step 1510, the KPI selector 1310 may select at least one attribute having the largest change contribution based on the model of each metric indicator. For example, using the model generated by the KPI modeler 1312, the derivative of each attribute can be taken to find the one with the largest derivative (change contribution). The attribute with the highest change contribution is the most significant or influential for that metric indicator.

[0126] In step 1515, the KPI selector 1310 can generate a set of metric indicators based on selecting a subset of metric indicators, where the subset is selected based on selecting the metrics with the largest statistically significant change. The metric with the largest entropy change can be determined by identifying whether the metric indicator may have an anomaly associated with it. As described above, one way to determine whether there is an anomaly in a metric indicator includes generating a dynamic dependency model using the time series data of the metric indicator. A first set of predictions of the values of the first variable or attribute of the metric indicator is generated based on the dynamic dependency model. The values from the model may be compared with the actual values to generate a statistical deviation using a statistical divergence test. The statistical divergence test includes, but is not limited to, the Kullback-Leibler information measure test. If the statistical deviation is significant (e.g., equivalent to two or more standard deviations), the KPI selector 1310 identifies the metric indicator as having some statistically significant information indicating that it is relevant or that one should look at it. This deviation indicates an anomaly, and the metric indicator with the anomaly is included in the set of metric indicators. In some embodiments, user personalization may also be used to add metric indicators to the set. For example, if the important KPI and relationship database 1332 has information that a particular user prefers some metric indicators for regular viewing, those KPIs may also be included in the set. The important KPI and relationship database 1332 can obtain this user-specific information from the clustering engine 1320.

[0127] In step 1520, the presentation manager can generate a graphical representation of each of the subset of metric indicators, each graphical representation including at least one attribute with the largest entropy change. Stated another way, the metric card can include information about the attributes that most affect the metrics identified by the KPI selector 1310 when each KPI is analyzed in 1510.

[0128] In some embodiments, the metric indicators are ordered based on statistical significance and / or practical significance. The ordering is based on the z-score (normalized standard deviation) distance between the observed distribution and the expected forecasted prediction, with the ranking in descending order. Practical significance is determined based on the product of how infrequently users within the enterprise probe anomalies in that area and how large the deviation is in terms of the z-score. Once the metric indicators are ordered and the metric indicators with the highest informational value are selected (subset), a graphical display of each of the ordered subset of metric indicators may be generated.

[0129] In step 1525, a graphical representation of the metric indicators, which may be ordered, is provided to a user's display for viewing.

[0130] While specific embodiments have been described, various modifications, variations, alternative configurations, and equivalents are possible. The embodiments are not limited to operation in any particular data processing environment, but operate freely in multiple data processing environments. In addition, while certain embodiments have been described using a particular sequence of transactions and steps, those skilled in the art will recognize that this is not limiting. While some flowcharts describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be permuted. A process may have additional steps not included in the figures. Various features and aspects of the above-described embodiments may be used individually or together.

[0131] Additionally, while particular embodiments have been described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are possible. Particular embodiments may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or any combination of different processors.

[0132] When a device, system, component, or module is described as being configured to perform a certain operation or function, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by executing computer instructions or code, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication; different pairs of processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0133] Specific details are provided in this disclosure to facilitate a thorough understanding of the embodiments. However, the embodiments may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the embodiments. This description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of other embodiments. Rather, the foregoing description of the embodiments will provide one skilled in the art with an enabling description for implementing various embodiments. Various changes may be made in the function and arrangement of elements.

[0134] Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. However, it will be apparent that additions, deletions, omissions, as well as other modifications and alterations can be made without departing from the broader spirit and scope set forth in the claims. Accordingly, while specific embodiments have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the claims.

Claims

Claim 1 A method implemented by a computer, comprising: For each metric indicator of a plurality of metric indicators of an enterprise, Generating a model of the metric indicator using machine learning, the model having each of a plurality of attributes that explain changes in the metric indicator as independent variables of the model, outputting a value of the metric indicator corresponding to the plurality of attributes, and the method further comprising, for each metric indicator of the plurality of metric indicators of the enterprise, Selecting at least one attribute having the largest change contribution to the value from the model among the plurality of attributes, the largest change contribution being calculated as the maximum derivative value of the at least one attribute, and the method further comprising, Obtaining a previous forecast of the current metric indicator; When the previous forecast is different from the value of the current metric indicator, selecting the current metric indicator as an element of a subset of the plurality of metric indicators, and the method further comprising, Generating a graphical user interface for each of the subset of the plurality of metric indicators, each graphical user interface of the subset of the metric indicators displaying information associated with the at least one attribute among the plurality of attributes, and the method further comprising, Providing the graphical user interface to the user's device for display to the user; The method further comprises, Recording the user's input for at least one metric indicator within the user interface; Generating a usage pattern of the user for the at least one metric indicator based on the recorded input; Generating a set of metric indicators including the at least one metric indicator based on the selection of the subset, the method comprising. Claim 2 Furthermore, Identifying a second subset of the plurality of metric indicators that the user can access; Selecting the subset of the plurality of metric indicators from the second subset of the plurality of metric indicators, the method according to claim 1. Claim 3 A method implemented by a computer, comprising: For each metric indicator of a plurality of metric indicators of an enterprise, Generating a model of the metric indicator using machine learning, the model having each of a plurality of attributes that explain changes in the metric indicator as independent variables of the model, and outputting a value of the metric indicator corresponding to the plurality of attributes, the method further comprising, for each metric indicator of the plurality of metric indicators of the enterprise, Selecting at least one attribute having the largest change contribution to the value from the model among the plurality of attributes, the largest change contribution being calculated as the maximum derivative value of the at least one attribute, the method further comprising Obtaining a previous forecast of the current metric indicator; and If the previous forecast is different from the value of the current metric indicator, selecting the current metric indicator as an element of a subset of the plurality of metric indicators, the method further comprising Generating a graphical user interface for each of the subset of the plurality of metric indicators, each graphical user interface of the subset of the metric indicators displaying information associated with the at least one attribute among the plurality of attributes, the method further comprising Providing the graphical user interface to the user's device for display to the user; The method further comprises Tracking usage patterns of the user interface by each of a plurality of users; and Identifying time-series data of the number of users viewing each of the plurality of metric indicators based on the usage pattern of the user interface; Generating a graphical trend card including the time-series data; and Providing the graphical trend card to one or more of the plurality of users in the user interface. **Claim 4** A program for causing a computer to execute the method according to any one of claims 1 to 3. **Claim 5** A system comprising a memory storing the program according to claim 4, and One or more processors for executing the program. **Claim 6** A memory device storing the program according to claim 4.

Images