Techniques for data-driven correlation of metrics

Abstract

To provide a method, a system, and a memory device for identifying a highly relevant content for a user to view in the form of key performance indicator (KPI) cards and providing a relevant view to the user automatically or by suggestion.SOLUTION: A method 1500 executed by a smart analysis system and / or smart analysis architecture, comprises: generating models of metric indicators having each attribute as an independent variable; selecting at least one attribute having a maximum change contribution based on the model of each metric indicator; generating a set of metric indicators based on selecting a subset of the metric indicators; generating a graphical representation of each subset of the metric indicators; and providing a graphical representation of ordered metric indicators to a user's display.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] REFERENCE TO RELATED APPLICATIONS This application is a continuation of a provisional patent application entitled "TECHNIQUES FOR DATA-DRIVEN CORRELATION OF METRIC" filed on May 31, 2019. Application No. 62 / 855,218, and "TECHNIQUES FOR DATA-DRIVEN CORRELATION OF METRIC" filed September 27, 2018. This application claims the benefit of and priority to Provisional Patent Application No. 62 / 737,518, entitled "Compounds for a Novel Microwave Oscillator," the entire contents of each of which are incorporated herein by reference in their entirety for all purposes. [Background technology]

[0002] background Metrics (also referred to herein as key performance indicators (“KPIs”)) provide useful information about a company's operating performance. It can be difficult to identify which metrics are most useful to a user in making decisions. Information relevant to a user can change quickly or stagnate. However, a user who regularly views only certain metrics may not notice significant or informative changes, especially if the user regularly views only stagnant metrics. Summary of the Invention

[0003] overview The techniques described herein provide practical and statistically useful information to a user by providing metrics that are informative, relevant, and personalized to that particular user. A system consisting of one or more computers can be configured to perform a particular operation or action by having software, firmware, hardware, or a combination thereof installed on the system and causing the system to perform an action during operation. One or more computer programs can be configured to perform a particular operation or action by including instructions that, when executed by a data processing device, cause the data processing device to perform an action. One general aspect includes a computer-implemented method for identifying and displaying highly informational content to a user, including: for each metric indicator of a plurality of metric indicators of a company, generating a model of the metric indicator, the model having each attribute of the plurality of attributes of the metric indicator as an independent variable of the model; for each metric indicator, selecting at least one attribute of the plurality of attributes having the largest entropy change contribution based on the model; generating a set of metric indicators based on selecting a subset of the plurality of metric indicators, the subset of the plurality of metric indicators being selected at least in part based on having a metric with the largest statistically significant change. A graphical representation of each of a subset of the plurality of metric indicators is generated, each graphical representation of the subset of metric indicators including at least one attribute of the plurality of attributes. The graphical representation is provided to a user device for display to a user. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0004] Implementations may include one or more of the following features: Optionally, multiple metrics Selecting a subset of metric indicators includes obtaining a previous time series model-based forecast of a current metric indicator, determining a current value of the metric indicator, and selecting the metric indicator if the previous forecast (expected value) is statistically significantly different from the current value. Optionally, generating a metric explanation with the largest entropy change for the subset of metric indicators includes applying the current values ​​of the metric indicators to a model along with values ​​of other independent metrics or variables to generate partial derivatives (rates of change) of each of the attributes (independent variables) in the model, and generating statistically significant entropy explanations using computational techniques including, but not limited to, locally interpretable model explanations and Shapley Additive Explanations. and selecting a subset of each of the attributes based on an ordered list of the size of the generated derivative, which is arbitrary, and how significant its contribution to the change in the metric. Optionally, the method may include recording user actions within the user interface, and generating user preference information for at least one metric indicator, frequency of use, and metrics viewed before and after each metric based on the recorded actions. Optionally, the set of metric indicators includes at least one metric indicator in the set. Optionally, generating a graphical display of each of the subset of metric indicators includes generating a metric card including at least one visual depiction of the at least one attribute. Optionally, usage behavior of a plurality of users using the user interface is tracked and used to provide suggestions for metrics of interest based on what other users, such as the current user, have viewed in the past. Optionally, trend data for each of the plurality of metrics based on the usage behavior of the plurality of users is generated. Optionally, a graphical trend card is generated including a visual depiction of the trend data. Optionally, the graphical trend cards are provided in the form of suggestions and recommendations of metrics of interest to one or more of the users in a user interface. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium. [Brief explanation of the drawings]

[0005] [Figure 1] 1 illustrates a high-level overview of a smart analytics system according to some embodiments. [Figure 2A] 1 illustrates an exemplary user interface with KPI cards of interest, according to some embodiments. [Figure 2B] 1 illustrates an example user interface for a drill-down view of a KPI card, according to some embodiments. [Figure 3]FIG. 1 illustrates an exemplary user interface with associated metric suggestions, according to some embodiments. [Figure 4] FIG. 1 illustrates a method for identifying related metrics according to some embodiments. [Figure 5] FIG. 10 illustrates another exemplary user interface with KPI cards of interest, according to some embodiments. [Figure 6] FIG. 10 illustrates another method for identifying related metrics according to some embodiments. [Figure 7] FIG. 10 illustrates an exemplary user interface for adding a KPI, according to some embodiments. [Figure 8] FIG. 10 illustrates an exemplary user interface for providing details of a new KPI, according to some embodiments. [Figure 9] FIG. 10 illustrates an example user interface for a drill-down view of new KPIs, according to some embodiments. [Figure 10] FIG. 1 illustrates an exemplary networked computer system, according to some embodiments. [Figure 11] 1 illustrates an exemplary cloud-based computer system, according to some embodiments. [Figure 12] FIG. 1 illustrates an exemplary computer system, according to some embodiments. [Figure 13] FIG. 1 illustrates an example architecture of a smart analytics system according to some embodiments. [Figure 14] FIG. 1 illustrates an exemplary method for generating metric recommendations, according to some embodiments. [Figure 15] FIG. 1 illustrates an exemplary method for displaying relevant content, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0006] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular inventive embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0007] Described herein are methods and systems for providing suggestions of metric indicators (also referred to as key performance indicators (“KPIs”)) for viewing in a user interface. Also described are techniques for providing highly informational content to a user without the user having to request a specific KPI. The user interface can provide a dashboard (also referred to as a cockpit) that can include visual depictions of metric indicators (e.g., business metric indicators or key performance indicators). The user interface can be provided by a computer system such as computer system 1200 of FIG. 12 , cloud infrastructure system 1102 of FIG. 11 , or server 1012 of FIG. 10 . The user interface can be provided to and useful to a user for viewing the metric indicators or the data underlying the metric indicators. Metric indicators can be data aggregations or snapshot views of the underlying data or time series trends created from that data with statistical transformations including, but not limited to, mean, median, standard deviation, quartiles, etc. For example, a metric indicator can provide trend data (e.g., sales have increased 10% over the past 60 days), snapshot information (e.g., year-to-date revenue is $10 million), etc. Business metrics can include values ​​such as headcount, revenue, attrition, inventory, etc. Metric indicators (e.g., key performance indicators) can be generated based on metrics. For example, a year-to-date revenue metric indicator can provide information about year-to-date revenue. Metric indicators can provide trend or historical information that helps a user understand a metric in some context. For example, a daily revenue value may not be as useful to a user as a quarterly revenue value.

[0008] As used herein, metric, metric indicator, variable, metric card, KPI, and KPI card are used interchangeably to describe data provided about and / or associated with a metric in a user interface. For example, KPI card 220a is a KPI card that shows information about a headcount metric. A collection of KPI cards is called a set. For example, a KPI I-cards 220a, 220b, 220c, 220d, 220e, 220f, 220g, and 215 are sets of KPI cards that are displayed to a user, as shown in FIG. 2A. In some embodiments, the KPI cards within a set are related. In some embodiments, a user may select KPI cards for display in a set that may or may not be related, in that there may or may not be metadata within the KPI cards that associates them with each other.

[0009] The described user interfaces may include, for example, user interfaces used by executives and their teams. Executives may include chief financial officers (CFOs), chief technical officers (CTOs), chief human resources officers (CHROs), chief executive officers (CEOs), departmental line managers, business analysts, etc. As described herein, the user interfaces may be referred to as dashboards. Optionally, the user interfaces may be used by any user.

[0010] When a business first starts using the described user interface, which may be provided as a cloud application, site install, or any other suitable implementation, the user interface for any given user may be generated based on the user's type, area of ​​interest, user role, and user responsibilities. For example, a CHRO may have a different view (e.g., different metric indicators shown) than a CEO. A CHRO may be more interested in employee data such as headcount, attrition, and diversity. However, a CEO may be more interested in revenue, inventory, production, etc. Thus, the out-of-the-box implementation may provide metric indicators that are more likely to be useful to a user based on the user's title or role within the business. Optionally, when installing the application, a user may select an initial dashboard configuration from several out-of-the-box default configurations, including those for the CEO, CHRO, CFO, CTO, etc.

[0011] As each user uses the user interface, the computer system can track and identify each user's usage patterns. Patterns can include initial viewing habits, sequential viewing habits, habits that can be identified as changing based on data values, etc. For example, a CHRO may develop a pattern of opening the user interface, viewing a headcount metric indicator, and viewing a hiring metric indicator if the headcount metric indicator indicates high headcount. However, if the headcount metric indicator is low, the CHRO may view an attrition metric indicator and view an attrition metric. These viewing habits and their sequence can be analyzed and stored. Details of this episodic memory storage are described in more detail with respect to FIG. 13.

[0012] Once a user's behavioral patterns are identified, the computer system may additionally use current data values ​​or trends to identify information that may be useful to the user based on the user's usage patterns. For example, a CHRO may be given a suggestion to view headcount metrics upon login because the headcount metrics are low. Optionally, the suggestion may include a notification along with the suggestion. For example, the suggestion may appear in the user interface as a pop-up containing a suggestion and notification such as, "Headcount has decreased since last month and is at a monthly low point. Would you like to see the headcount metrics?" The suggestion and notification wording is clickable so that the user can click the text to view the headcount metrics within the user interface. In some embodiments, relevant data is provided to the user without requiring a user preference simply because it is statistically significant or because it is relevant for a particular domain.

[0013] Additionally, all user usage behaviors, user input, and metric indicator behavior over time can be used to identify relationships between metric indicators. For example, a CHRO's behavior of viewing an attrition metric indicator shortly after (within a threshold time period) a headcount metric indicator may cause a computer system to identify a relationship between the attrition metric indicator and the headcount metric indicator. Similarly, a CHRO's behavior of viewing an employment metric indicator shortly after (within a threshold time period) a headcount metric indicator may cause a computer system to identify a relationship between the employment metric indicator and the headcount metric indicator and place a weight on that edge in a graph of relationships where the strength of the relationship is defined by the weight on the edge and the nodes are metrics. Metric indicators may have associated metadata that the computer system can use to create a relationship between two metric indicators. Optionally, a metric indicator can have any number of relationships. In the above example, the headcount metric indicator has two associated metric indicators, and the attrition metric indicator and the employment metric indicator each have one. Optionally, the computer system may identify relationships by analyzing the user logs to find indicators that are viewed close in time based on timestamps in the user logs. In some embodiments, the user logs of multiple users can be used to generate relationships between the metric indicators used for all other users.

[0014] Metric indicator behavior over time can also be used to identify relationships between metric indicators. For example, a computer system can analyze the values ​​(trends) of metric indicators over time. The computer system analysis method can, for example, identify leading and lagging indicators. One example is that when inventory starts to trend downward, revenue also trends downward, but sometimes later (e.g., three days, two weeks, one month, etc.). The computer system can analyze the similarity of trends with a time lag for revenue trends and identify an inventory metric indicator as a leading indicator for the revenue metric indicator, and conversely, identify a revenue metric indicator as a lagging indicator for the inventory metric indicator. Metadata associated with each metric indicator can be used to store the relationship between two metric indicators and the type of relationship (leading indicator / lagging indicator) between them.

[0015] In some embodiments, a user can input relationships between metric indicators via a user interface. Because metric indicators can be created by users, inputting relationships within metric indicator information can help other users create relationships.

[0016] Relationships created using user logs, explicit user input of relationships via a user interface, and / or metric indicator behavior can be used by a computer system to generate recommendations for a user based on the user's role (e.g., HR team member, finance team member, etc.), user behavior, and / or the user's current view, as described in more detail below.

[0017] One embodiment includes a technique for crowdsourcing metric indicators. In some embodiments, the system may be a crowdsourced system provided as a site install or as a software as a service ("SaaS") implementation. When initially set up for a business, which may be as a cloud-based system, the out-of-the-box dashboard for any given user may contain a set of metric indicators that are relevant to that type of user. For example, The CEO, for example, may have different metric indicators on the dashboard than the CHRO. One of the metric indicators may be a trending metric indicator. The trending metric indicator may provide the trending metric indicator, for example, using a tag cloud. For example, the trending metric indicator may include metric indicators being viewed by other users in the company, or in some embodiments, more specifically by others on the user's team. In some embodiments, when the installation is new, trending metric indicators may be identified based on what other users at other companies are viewing as common domain knowledge within the company. The tag cloud may provide a list of metric indicator names with varying font sizes based on the popularity of the metric indicator. For example, a metric indicator that is frequently viewed may have a larger font than a metric indicator that is viewed by only one user. In some embodiments, the trending metric indicator may be configured by the user to identify a tag cloud based on the user's team, the company in general, or other similar companies hosted on the same cloud-based system, regardless of how long the business has used the system. The tag cloud may be generated by tracking user behavior and based on which metric indicators are being viewed at any given time. The trending metric indicators can be updated periodically, for example, hourly, or upon refresh by a user. The metric indicators listed in the trending metric indicators can be selected by a user. Upon selection, the metric indicator can be added to the user's dashboard, for example. In some embodiments, selecting a metric indicator name from the tag cloud can provide a details screen that provides information associated with the selected metric indicator.

[0018] In some embodiments, a metric indicator includes metadata used to model a business metric. Each metric indicator can have several attributes. Each metric indicator definition can describe a primary value along with a detailed perspective that provides additional information about the primary value.

[0019] In some embodiments, the metric indicators are used across various business domains (e.g., customer experience (“CX”), human capital management (“HCM”), enterprise resource planning (“ERP”), and the like). ), and supply chain management (SCM) Each metric indicator can be informed by persona needs and business goals within their specific domain. The user interface can provide immediate value and morph over time to best serve the business. Thus, the metric indicators can reflect the prior knowledge of the business and "learn" over time. The user interface can morph (change), for example, because new metric indicators can be used as defaults for users in the user interface and different attributes for each metric indicator might be shown. Additionally, suggestions can be provided that allow the user to modify the details of the dashboard / user interface.

[0020] At initial installation (e.g., site install or cloud-based implementation), two or more metric indicators can be defined and provided as part of the application. After installation, metric indicators can continue to be defined organically and picked up (selected for use) by users. Metric indicators can also arise organically over time to surface new or changing business needs. Metric indicators can reflect prior knowledge of the business and "become" selected over time as recommendations from user usage history. Metric indicators can leverage data to model a business. Data can be provided by industry benchmark data, third-party data sources, sales and production data sources, etc.

[0021] 1, a high-level block diagram illustrates a smart analytic system 100. The smart analytic system 100 can be used to generate metric indicators (KPIs and cards) for presentation to a user. The smart analytic system 100 includes a user device 105, a smart analytic server 110, a smart analytic database 115, and an enterprise data source 120. The smart analytic system 100 can include additional components not shown in FIG. 1 for the sake of brevity.

[0022] User device 105 may be any suitable user computing device, such as user device 1002, 1004, 1006, or 1008 of Figure 10, client device 1104, 1106, 1108 of Figure 11, or computer system 1200 of Figure 12. Using user device 105, a user can access user interface 125 of smart analytics server 110 to obtain a graphical user interface showing KPI cards and suites.

[0023] The smart analytic server 110 can be a computer system such as the computer system 1200 of FIG. 12 or the server 1012 of FIG. 10. The smart analytic server 110 includes a user interface 125 and a KPI subsystem 130. The user interface 125 can be used by the user device 105 to gain access to information generated by the KPI subsystem 130. The KPI subsystem 130 generates metric indicators (KPIs), generates metric indicator recommendations, and otherwise identifies high-value content for the user to view, as described in more detail in FIG. 13. The KPI subsystem 130 accesses the smart analytic database 115 to store and retrieve information generated by the smart analytic server 110. The smart analytic database can include a metadata index 1330, a key KPI and relationship database 1332, a role and KPI database 1334, and an action knowledge base 1336 of FIG. 13.

[0024] The enterprise data source 120 can be any data source of the enterprise that provides data used to create the KPIs. This information can be retrieved from the enterprise data source 120 by the smart analytics server 110 to perform metric indicator analysis, as described in more detail below.

[0025] FIG. 2A shows an example dashboard 200 for a CHRO. A user can log in to the dashboard / user interface, which can be hosted on a computer system such as, for example, computer system 1200 of FIG. 12, cloud infrastructure system 1102 of FIG. 11, or server 1012 of FIG. 10. Dashboard 200 can include multiple metric indicator cards 215 and 220. Cards 215 and 220 can each provide a view of activity represented by metric indicators. Metric indicator cards 215 and 220 can each represent information about a particular metric. The information may be stored in a data store of information for the enterprise. Each user can group various metrics on the metric indicator cards into one or more sets. In some embodiments, the sets are created by the system or grouped based on the underlying metadata of each of the metric indicator cards 215 and 220. Each metric indicator card can display information about other related metric indicator cards (e.g., identification). The metric indicator cards 215 and 220 may each be created by a user of the system, created by the system based on user behavior, or provided with the initial implementation of the system.

[0026] Metric indicator cards 215 and 220 may be generated by a user by selecting metrics to view along with metadata. These cards may be personalized by each user while still relying on the same underlying data. For example, metric indicator cards 215 and 220 may show line graph trends for one user, while another user prefers a pie chart display. These types of user-specific personalizations are stored in metadata for that user within the metric indicator cards. Additional information in the metadata of metric indicator cards 215 and 220 includes relationship data regarding the associated metric indicator. The relationship data may be generated manually, by rules, and / or using machine learning.

[0027] Dashboard 200 may include a section that provides notifications 205. Notifications 205 may include team notifications applicable to all users in a team. Notifications 205 may provide information about specific users (e.g., Linda and Thomas) that is useful to all members of the team. Alerts 210 may provide relevant suggestions and notifications to the user. For example, alert box 210 includes an alert that new job attrition is at 12%. New job labor turnover rate metric indicator 215 shows the user further data related to this alert. A second alert 210 also states that the top quartile of employees in U.S.-based manufacturing attrition is 4.8 times higher than the expected rate, which may be useful information to a user in a CHRO user interface. Alerts are clickable so that selecting the alert can provide a view of metric indicators that provide relevant data about the attrition information.

[0028] If an alert is selected, the user interface 250 of FIG. 2B may be provided to the user. The user interface 250 may include information relevant to the alert. For example, the user interface 250 may include further insights 255 into data relevant to high attrition rates. The leading indicator 260 may provide the user with further insights regarding high levels of attrition and employee departures. The high risk metric indicator 265 may identify parts of the company that are at risk for additional attrition based on historical data. The loss metric indicator 270 may provide information about employees who left in the most recent quarter. The attrition metric indicator 275 may provide monthly attrition information based on region (e.g., Europe and Africa). The attrition by department metric indicator 280 may provide quarterly information on attrition organized by department.

[0029] Dashboard 250 may be one view of a drill-down of metric indicators based on a selected metric indicator from initial dashboard 200. In some embodiments, different metric indicators may be configured to be displayed based on the selected metric indicator.

[0030] 3 may be another example dashboard 300. For example, a finance team member may be viewing an income statement metric indicator that includes an income statement graph 305. A user may select a revenue bar in the income statement graph 305, which may display a revenue metric indicator 310. The revenue metric indicator 310 may also display a direct revenue metric indicator, an indirect revenue metric indicator, a sales velocity metric indicator, a revenue-to-Amazon metric indicator, and a sales metric indicator. For relationships identified between, etc., metric indicators in the associated smart indicator selection box 315 may be displayed for user selection and viewing.

[0031] In some embodiments, the metric indicators shown in the associated smart indicators selection box 315 may be selected based on relationships created based on one or more users' usage behavior (e.g., analysis of the users' user logs), explicit relationship creation by the user through a user interface, and / or automatic relationship creation based on a computer system analyzing metric indicator behavior over time. In some embodiments, the viewing user's behavior may be used to select the metric indicators shown in the associated smart indicators selection box 315. For example, if a user has recently viewed a metric indicator, it may be excluded (or included) in the associated smart indicators selection box 315.

[0032] FIG. 4 may be a flowchart 400 of a method for providing data-driven correlation of metrics. The method of flowchart 400 may be executed by a computer, such as, for example, server 1012 of FIG. 10 , cloud infrastructure system 1102 of FIG. 11 , or computer system 1200 of FIG. 12 . Flowchart 400 may begin with step 405 of providing a dashboard to a user via a user interface, the dashboard including a visual depiction of metric indicators, where the dashboard is configurable for each user. For example, user interface 200 or 250 provides metric indicators for human resource (“HR”) users. Different types of users may want to view financial information (e.g., revenue metric indicators) and can configure their dashboards accordingly. In some embodiments, initial dashboard configurations may be provided to various types of users. For example, an HR user may select an HR dashboard configured with headcount metric indicators, hiring metric indicators, attrition metric indicators, etc. As another example, a finance user may select a finance dashboard configured with revenue metric indicators, sales metric indicators, etc. Each user can then remove and add metric indicators to make the dashboard unique to that user. The method can continue at step 410 by tracking the user's usage behavior in the user interface. For example, an HR user will typically see a metric indicator for attrition rates for the most recent quarter in the United States when the user logs in. This can be identified as a usage pattern. Each user activity may be stored in a user log. The user log may include, for example, the user name, the activity, and a timestamp. The method can continue at step 415 by identifying a relationship between the first metric indicator and the second metric indicator.For example, user log information may indicate that a user views a first metric indicator and a second metric indicator within a threshold time period of each other. The method may continue with step 420 of providing a suggestion to view a second metric indicator when the first user is viewing the first metric indicator on the dashboard. For example, the user interface 300 provides an exemplary view with the suggestions. The related smart metric indicator selection box 315 provides a suggestion of a related metric indicator based on the user viewing the revenue metric indicator 310.

[0033] Returning to step 415, identifying a relationship between a first metric indicator and a second metric indicator may be performed by a user, for example, via a user interface. The relationship may include identifying relationships from manually entered relationships. A user may be able to tag metric indicators as related and mark them. For example, a user may select two metric indicators and mark the first metric indicator as leading (i.e., the trend of this metric indicator leads the trend of the second metric indicator) and the second metric indicator as lagging (i.e., the trend of this metric indicator follows the trend of the second metric indicator). Metadata stored with the metric indicators enables this tagging and relationship creation. The metadata may be stored in a data repository 1014, for example, as described with respect to FIG. 10.

[0034] In step 415, identifying a relationship between the first metric indicator and the second metric indicator may include identifying the relationship based on a user log that records user behavior and activity in a user interface. For example, a sample usage log is shown in Table 1.

[0035] [Table 1]

[0036] Using the logs shown in Table 1 and using association rule learning-based data mining, metric indicators that are frequently viewed together are identified, and associations are made between those metric indicators. This analysis may identify the likelihood of a relationship without identifying the type of relationship. For example, using the example logs in Table 1, "Gross Margin" and "Revenue by Region" may be identified as having a relationship because multiple users execute them sequentially during a session. An association rule may identify that the metric indicators were executed by the same user in the same session or within a specific time period by the same user. The sequence of execution may provide insight into the association rule algorithm. The association may include additional information, including the type of user (user role and / or domain) to which the relationship applies. Using the above example, the user's domain may be CFO and the user's role may be Finance Officer. User logs may be collected for each user within the company.

[0037] In step 415, identifying a relationship between the first metric indicator and the second metric indicator may include automatically identifying the relationship through analysis of metric indicator trends. The metric indicator value trends may be used to identify the related metric indicator and the type of relationship (e.g., leading / lagging metric indicator). Periodically (e.g., daily, weekly, monthly, quarterly, yearly, etc.), snapshots of the values ​​of each metric indicator may be taken. The snapshots may be used to identify any trends, patterns, and / or interdependencies. For example, if a first metric indicator increases, then a second metric indicator also increases, and this trend is consistently shown, a relationship can be inferred between the two metric indicators. The time period used to identify whether two metric indicators trend together or one leads / lags the other can vary. For example, some metric indicators may trend together and be seen in each snapshot, while in other relationships, a second metric indicator may lag another metric indicator by several months. As another example, if a third metric indicator increases, then a fourth metric indicator decreases, a relationship can be inferred between the metric indicators. Table 2 provides example metric indicator data snapshot values.

[0038] [Table 2]

[0039] Table 2 provides a snapshot of values ​​collected and stored for three metric indicators (gross margin, revenue, and expense) over a two-day period. Such metric indicator snapshots can be collected across all hosted customers in a cloud-based hosted environment. Algorithms can be applied to the snapshot data to identify trends in the metric indicator values. Correlations between metric indicator values ​​that consistently trend together (e.g., over time, both metric indicators increase and / or decrease together, or one metric indicator increases followed by an increase and / or decrease some time later) are used to identify that the metric indicators are related and the type of relationship (e.g., leading / lagging).

[0040] 5 shows another exemplary user interface 500, which may be, for example, an initial, out-of-the-box dashboard provided to a user. As the user engages the user interface, the tracking described above may be used to provide suggestions based on the user's usage patterns and current data associated with those usage patterns.

[0041] The user interface 500 may be, for example, a dashboard for a CEO. The CEO user interface 500 may include metric indicators such as quarterly revenue 505, year to date (“YTD”) revenue 510, etc. The CEO user interface 500 may also include trending metric indicators 515. The trending metric indicators 515 may be a tag cloud that provides trending metric indicator information (also described herein as crowd-sourced metric indicator information). More popular metric indicators are shown in larger fonts. In the user interface 500, for example, the employee diversity KPI is currently displayed and is the most popular metric indicator, as indicated by the largest font. In some embodiments, color coding may further be used to highlight the most popular / currently trending metric indicators. For example, bright colors may be used for the most trending ones, while dimmer colors, grays, or other less noticeable colors may be used for the most popular ones. Different color coding may be used for metric indicators that are less popular at the time trend information is collected. As a user views user interface 500, trend metric indicators 515 may change as the metric indicator's trend changes. For example, as a profit KPI becomes more popular (used by others at the company, team, etc.), the font size of "profit" in trend metric indicator 515 may increase. Similarly, for example, as an employment rate KPI becomes less popular (viewed by fewer users at the company, team, etc.), the font size of "employment rate" in trend metric indicator 515 may decrease. In some embodiments, a user may select a metric indicator in trend metric indicator 515 by, for example, clicking on the metric indicator name in trend metric indicator 515. Selecting a metric indicator name may drill down into the selected metric indicator by displaying a more detailed screen with the metric indicator and / or other underlying information about the metric indicator. In some embodiments, the user can select whether trend metric indicators 515 analyzes metric indicator trends based on all users within an enterprise, users belonging to a particular team within an enterprise, users belonging to a particular team across other enterprises if information about other enterprises is available, etc.

[0042] FIG. 6 shows a flowchart 600 illustrating a method for crowdsourcing metric indicators. The method may be performed by, for example, the server 1012 of FIG. 10 , the cloud infrastructure system 1102 of FIG. 11 , or the computer system 1200 of FIG. 12 . The method may include, at step 605, providing a plurality of metric indicators in a user interface. In some embodiments, the metric indicators are all generated by a software manufacturer. In some embodiments, the metric indicators may be generated and / or customized by a user. The method may further include, at step 610, tracking a plurality of users using the user interface. In some embodiments, the users may be teammates of the individual using the user interface. In some embodiments, the users may be other users within the individual using the user interface's company. In some embodiments, the users may be from other companies (enterprises). The method may further include, at step 615, identifying trending data for each of the metric indicators based on tracking user usage. For example, if a large number of users (e.g., 15 users in a 40-user company) are viewing a metric indicator, the metric indicator may be considered trending. Determining whether a metric indicator is trending upward or downward may include analyzing whether the number of users viewing the metric indicator is increasing or decreasing over a period of time (e.g., 30 minutes, 24 months, 3 quarters, 13 weeks). Additionally, the number of users viewing a first metric indicator can be considered in relation to the number of users viewing other metric indicators to determine whether the first metric indicator is popular.Additionally, the total number of users in the system may further indicate, for example, the popularity of a metric indicator based, for example, on the percentage of users viewing the metric indicator (e.g., 90% of users viewing the metric indicator may indicate high popularity and large font in the tag cloud, while 2% of users viewing the metric indicator may indicate low popularity and no metric indicator in the tag cloud of trending metric indicators 515). A metric indicator with less attention and / or views may have a low trend value. A metric indicator with decreasing views may also have a low trend value. The method also includes, in step 620, suggestions for viewing these KPIs. The method may also include providing trending data to one or more users in a user interface through recommendations and suggestions. For example, a tag cloud may be generated that identifies trending metric indicators, such as trending metric indicator 515 of FIG. 5. The tag cloud may provide a clickable list that shows metric indicator names and provides information about the trending value or popularity of the named metric indicator. For example, the most popular metric indicators may be displayed in a larger font in the clickable list. Optionally, the method may include receiving a selection of a named metric indicator in the clickable list and, in response to the selection, providing the user, via the user interface, a view of details of the selected metric indicator.

[0043] Returning to Figure 5, user interface 500 may include an option to add a metric indicator 520. Selecting add metric indicator 520 may provide the user interface shown in user interface 700 of Figure 4. User interface 700 may provide a screen 705 that allows the user to select the type of selected metric indicator.

[0044] FIG. 8 may show a user interface 800 that may be used to enter data to be associated with a metric indicator. The base, interaction, and relationship, when entered by a user, provide metadata to be associated with the metric indicator. For example, a target persona may identify the type of user to target. As another example, associated metric indicators may be identified to cross-reference options (to create relationships for recommendations, as described with respect to FIGS. 3 and 4). Each of the metadata values ​​may further be used to identify relevant results when a user searches within the user interface.

[0045] Within the user interface, "new" metric indicators can be data-driven based on the needs of the consuming persona and can reflect any number of Oracle and non-Oracle data sources. New metric indicators are user-configurable and can be viewed across multiple touchpoints based on user preferences. New metric indicators can be driven by business insights derived from configurable thresholds, industry benchmarks, or advanced intelligence ("AI")-driven monitoring. Metric indicators can be pushed (alerts) to users based on business urgency. Metric indicators can further serve as a focal point for applications. Metric indicators can also be configurable by the user. Metric indicators can be viewed across multiple touchpoints based on the user. Metric indicators can filter all available metric indicators based on a range of parameters, not just the metric indicators currently shown on their user interface. Search can provide a mechanism for simple and complex searching / filtering. Easy controls can be provided for filtering and surfacing sets of metric indicators.

[0046] In some embodiments, search parameters may include, for example, thematic / business objectives, KPI status and status severity, usage / trends, seasonality, etc. In some embodiments, search may always be present on the user interface. Search may be integrated with the Ask feature. Additionally, search may not be based on derived relationships.

[0047] In some embodiments, the system can utilize metadata tags and relationships derived from application usage as described with respect to FIGS. 3 and 4. .

[0048] The application / user interface can be driven by a consistent definition of metric indicators that can be configured to be surfaced as desired by the user. One example of configuring metric indicators can include setting personalized thresholds that override metric indicator defaults. Threshold types can include personal thresholds, benchmarks, and automatic / AI detection. Metric indicators can further include custom navigation, multiple thresholds, override directionality, custom navigate to subsequent processes, generic rules, override rules, and / or custom tags. Based on user preferences regarding thresholds and update mechanisms defined in the user interface for metric indicators, the user interface and / or server system can provide those updates. Users can have the ability to configure how notifications of different severities are sent, e.g., text, email, voice, etc., to mobile devices. Where possible, notifications can also be accompanied by links back to the original metric indicator and next step actions.

[0049] Within the user interface, metric indicators can provide a snapshot into the business and a starting point for many subsequent actions. These next steps are intended to further describe or explore the business and take action on problems or opportunities revealed by the metric indicators. Next steps span a continuum from tightly defined automated drill-downs to fully customizable navigation paths. Ultimately, metric indicators can effectively act as a gateway to a wealth of capabilities.

[0050] In some embodiments, drilling into details may be available. FIG. 9 is an example metric indicator details view 900 for a monthly revenue metric indicator. For example, based on the metadata, including derived data, and the metric indicator details, a selection of related indicators may be selected for the monthly revenue metric indicator drill-down menu. Drill-down analysis is automatically generated and mapped to predefined patterns. The optimal pattern can be automatically selected based on the metric indicator type (e.g., focused on time, leading indicators, aggregate information requiring disaggregation, etc.) and the set of metric indicator attributes. This selection may also be overridden at the metric indicator definition level. Even with the default selection, other drill-down patterns are still indicative, and users can still see them if they select them.

[0051] Multiple attributes of metric indicators themselves and observations of how they are used can be used to drive the automated generation of insightful analytical experiences, also known as drill-down screens. This approach removes the burden of manually creating content from users and reveals insights that would otherwise be missed. First, metric indicator metadata and metric indicator type definitions can directly determine and populate drill-down detail screens. One example could be rendering a range of information permutations captured in a metric indicator to present users with information broken down by dimension or expanded by prompts. Second, drill-down can also be tailored based on user preferences (thresholds). Personal thresholds can inform relevance for specific users and can also be aggregated across user populations. Data captured around user behavior and system usage provides a third input. Tailored drill-down detail templates can be tailored to specific metric indicators, related metric indicators based on usage, leading / lagging indicators, related indicators, etc. The detailed view may include factors that contribute to, are associated with, or explain the analyzed and other factors. The detailed view may span or incorporate multiple metric indicators. Fourth, derived insights from machine learning algorithms (which identify patterns or anomalous events across large amounts of data) may also be collected from system data and system usage, attached to metric indicators, and surfaced in these metric indicator drill-downs. Classic (TIME) Historical & Predictive Analysis , to inform the state of the business. Because this is a dynamic system with constantly changing usage, data, and content, the output morphs over time based on the state of the application. Correlated metric indicators, leading / lagging indicators, or predictive analytics are good examples.

[0052] Some embodiments may include a method for providing a metric indicator authoring approach for dynamic analysis. The method may be implemented by a computer system. The method may include providing a user interface to a user. The method may further include receiving, via the user interface, a definition of the metric indicator, including metadata associated with the metric indicator. For example, selecting the save button in FIG. 8 may transmit a value entered via the user interface to the computer system. The method may further include generating a metric indicator detail view including data associated with the metric indicator based on the metadata. The detail view may be similar to, for example, detail view 900 in FIG. 9. The method may further include providing the detail view via the user interface.

[0053] The infrastructure may be implemented in a variety of different environments, including a cloud environment (which may be various types of cloud, including private cloud environments, public cloud environments, and hybrid cloud environments), an on-premise environment, a hybrid environment, etc.

[0054] 10 shows a simplified diagram of a distributed system 1000 for implementing an embodiment. In the illustrated embodiment, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to a server 1012 via one or more communication networks 1010. The client computing devices 1002, 1004, 1006, and 1008 may be configured to run one or more applications.

[0055] In particular embodiments, server 1012 may provide services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, such as in a software-as-a-service (SaaS) model, to users of client computing devices 1002, 1004, 1006, and / or 1008. Users operating client computing devices 1002, 1004, 1006, and / or 1008 may utilize the services provided by these components by interacting with server 1012 utilizing one or more client applications.

[0056] 10, the server 1012 may include one or more components 1018, 1020, and 1022 that implement the functions performed by the server 1012. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. It should be appreciated that a wide variety of system configurations are possible that may differ from system 1000. Thus, the embodiment shown in Figure 10 is an example of a distributed system for implementing the system of the embodiment and is not intended to be limiting.

[0057] Users use client computing devices 1002, 1004, 1006, and / or 1008 to run one or more applications, which may generate and then process one or more storage requests according to the teachings of this disclosure. A client device may provide an interface that allows a user of the client device to interact with the client device. The client device may also output information to the user via this interface. Although FIG. 10 shows only four client computing devices, any number of client computing devices may be supported.

[0058] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may include various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems, various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, Google Chrome® OS, including Palm OS®). Portable handheld devices may include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices (e.g., Microsoft Xbox® gaming consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.). Client devices may run a wide variety of applications, such as various internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), etc. The device may be capable of running various applications and may use various communication protocols.

[0059] Network 1010 may be any type of network known to those skilled in the art that is capable of supporting data communications using any of a variety of available protocols, including TCP / IP (transmission control protocol / Internet protocol), SNA (systems network architecture), IPX (Internet packet exchange: Internet packet exchange), AppleTalk (registered trademark), etc. By way of example only, network 1010 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide-area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PST), or the like. N:public switched telephone network), infrared network, wireless network ( For example, the Institute of Electrical and Electronics Engineers (IEEE) 802 This may include wireless networks operating under any of the .11 protocol suite, Bluetooth® and / or any other wireless protocol, and / or any combination of these and / or other networks.

[0060] The server 1012 may be one or more general-purpose computers, dedicated server computers (for example, PC (personal computer) servers, UNIX (registered trademark) The servers 1012 may be configured in various configurations and / or combinations, including servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable configurations and / or combinations. The servers 1012 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage that can be virtualized to maintain virtual storage for the servers. In various embodiments, the servers 1012 may be adapted to run one or more services or software applications that provide the functionality described in the above disclosure.

[0061] The computing systems within server 1012 may run one or more operating systems, including any of the operating systems described above, as well as commercially available server operating systems. Server 1012 may also run any of a variety of other server and / or middle-tier applications, including a hypertext transport protocol (HTTP) server, a file transfer protocol (FTP) server, a common gateway interface (CGI) server, a JAVA server, a database server, etc. Exemplary database servers are commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), etc. This includes, but is not limited to:

[0062] In some implementations, server 1012 may include one or more applications for parsing and consolidating data feeds and / or event updates received from users of client computing devices 1002, 1004, 1006, and 1008. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party sources and continuous data streams that may include real-time events related to sensor data applications, financial stock tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 1012 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 1002, 1004, 1006, and 1008.

[0063] The distributed system 1000 may also include one or more data repositories 1014, 1016. In particular embodiments, these data repositories may be used to store data and other information. For example, one or more of the data repositories 1014, 1016 may be used to store information, such as data used to analyze and display on a dashboard. The data repositories 1014, 1016 may be in a variety of locations. For example, a data repository used by the server 1012 may be local to the server 1012 or remote from the server 1012, and may be managed by the network. The data repositories 1014, 1016 communicate with the server 1012 via a network-based connection or a dedicated connection. The data repositories 1014, 1016 may be of different types. In particular embodiments, the data repositories used by the server 1012 may be databases, for example, relational databases such as those provided by Oracle Corporation® and other manufacturers. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to SQL-formatted commands.

[0064] In particular embodiments, one or more of the data repositories 1014, 1016 may be used by an application to store application data. The data repositories used by an application may be of various types, such as, for example, a key-value store repository, an object store repository, or a general-purpose storage repository supported by a file system.

[0065] In particular embodiments, the storage-related functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 11 is a simplified block diagram of a cloud-based system environment 1100 that may provide various storage-related services as cloud services, according to particular embodiments. In the embodiment shown in FIG. 11, a cloud infrastructure system 1102 may provide one or more cloud services that users may request using one or more client computing devices 1104, 1106, and 1108. The cloud infrastructure system 1102 may include one or more computers and / or servers, which may include those described above with respect to server 1012. The computers in the cloud infrastructure system 1102 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0066] The network 1110 may facilitate communication and exchange of data between the clients 1104, 1106, and 1108 and the cloud infrastructure system 1102. The network 1110 may include one or more networks. The networks may be of the same type or different types. The network 1110 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0067] The embodiment shown in Figure 11 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in other embodiments, cloud infrastructure system 1102 may have more or fewer components than those shown in Figure 11, may combine two or more components, or may have components in a different configuration or arrangement. For example, while Figure 11 shows three client computing devices, in alternative embodiments, any number of client computing devices may be supported.

[0068] The term cloud service is generally used to refer to services made available to users on demand over a communications network, such as the Internet, by a service provider's system (e.g., cloud infrastructure system 1102). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are distinct from the customer's own on-premise servers and systems. The cloud service provider's systems are managed by the cloud service provider. Thus, customers can access the cloud service provider's services without having to purchase separate licenses, support, or hardware and software resources for the services. A cloud service provider may also use cloud services offered by other providers. For example, a cloud service provider's system may host an application, and users may order and use the application on demand over the Internet without having to purchase infrastructure resources to run the application. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, and Java® cloud services, are offered by Oracle Corporation® of Redwood Shores, California.

[0069] In certain embodiments, the cloud infrastructure system 1102 may be a hybrid service model, including a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, a Service Infrastructure as a Service (IaaS) model, etc. A cloud infrastructure system 1102 may provide one or more cloud services using a variety of models. The cloud infrastructure system 1102 may include a suite of applications, middleware, databases, and other resources that enable the delivery of various cloud services.

[0070] The SaaS model allows applications or software to be delivered as a service to customers over a communications network such as the Internet, without the customer having to purchase the hardware or software for the underlying application. For example, the SaaS model may be used to provide customers with access to on-demand applications hosted by the cloud infrastructure system 1102. Examples of SaaS services offered by Oracle Corporation® include services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, and social applications. This includes, but is not limited to,

[0071] The IaaS model is commonly used to provide flexible computing and storage capabilities by providing infrastructure resources (e.g., servers, storage, hardware, and networking resources) to customers as cloud services. Various IaaS services are offered by Oracle Corporation (registered trademark).

[0072] The PaaS model is commonly used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without having to procure, build, or manage the environment resources. Examples of PaaS services offered by Oracle Corporation are Oracle This includes, but is not limited to, Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0073] Cloud services are generally provided on an on-demand, self-service basis, on a subscription basis, and in a flexible, scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by cloud infrastructure system 1102 via a subscription order. Cloud infrastructure system 1102 then processes the order to provide the customer with the services. Provide the services requested in the customer's subscription order. Cloud infrastructure system 1102 may be configured to provide one or more cloud services.

[0074] Cloud infrastructure system 1102 may provide cloud services through a variety of deployment models. In a public cloud model, cloud infrastructure system 1102 may be owned by a third-party cloud service provider, and cloud services are offered to general public customers. These customers may be individuals or businesses. In certain other embodiments, under a private cloud model, cloud infrastructure system 1102 may function within an organization (e.g., within a corporate organization), and services are offered to customers within the organization. For example, these customers may be various departments within a company, such as the human resources department, payroll department, or individuals within the company. In certain other embodiments, under a community cloud model, cloud infrastructure system 1102 and the services it offers may be shared among various organizations within an associated community. Various other models, including hybrids of the above models, may also be used.

[0075] Client computing devices 1104, 1106, and 1108 may be of different types (e.g., devices 1002, 1004, 1006, and 1008 shown in FIG. 10) and may be capable of operating one or more client applications. Users may use the client devices to interact with cloud infrastructure system 1102, such as to request services provided by cloud infrastructure system 1102.

[0076] In some embodiments, the processing performed by cloud infrastructure system 1102 may include big data analytics. This analytics may involve using large data sets, analyzing, and processing them to detect and visualize various trends, behaviors, relationships, etc. within this data. This analytics may be performed by one or more processors, possibly processing the data in parallel, running simulations with the data, etc. The data used for this analytics may be structured data (e.g., data stored in a database or structured according to a structural model) and / or unstructured data (e.g., data blobs (binary large objects)). The object may include

[0077] 11 , cloud infrastructure system 1102 may include infrastructure resources 1130 utilized to facilitate the provision of various cloud services offered by cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, etc.

[0078] In particular embodiments, to facilitate efficient provisioning of these resources to support the various cloud services offered by cloud infrastructure system 1102 to different customers, resources may be organized into sets of resources or resource modules (also referred to as "pods"). Each resource module or pod may include a pre-integrated, optimized combination of one or more types of resources. In particular embodiments, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, and a second set of pods may be provisioned for Java services, etc., which may include a different combination of resources than the pods in the first set of pods. For some services, the resources required to provision these services may be configured separately. The resources allocated to a service may be shared among services.

[0079] Cloud infrastructure system 1102 itself may use services 1132 internally that are shared by different components of cloud infrastructure system 1102 and that facilitate the provisioning of services by cloud infrastructure system 1102. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services enabling cloud support, email services, notification services, file transfer services, etc.

[0080] Cloud infrastructure system 1102 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in FIG. 11 , the subsystems may include a user interface subsystem 1112 that allows users or customers of cloud infrastructure system 1102 to interact with cloud infrastructure system 1102. User interface subsystem 1112 may include a variety of different interfaces, such as a web interface 1114, an online store interface 1116 through which cloud services offered by cloud infrastructure system 1102 are advertised and available for consumer purchase, and other interfaces 1118. For example, a customer may use a client device to request one or more services (service request 1134) offered by cloud infrastructure system 1102 using one or more of interfaces 1114, 1116, and 1118. For example, a customer may access an online store, browse cloud services offered by cloud infrastructure system 1102, and place a subscription order for one or more services offered by cloud infrastructure system 1102 and for which the customer wishes to subscribe. The service request may include information identifying the customer and one or more services for which the customer desires to subscribe. For example, a customer may submit an order to subscribe to storage-related services provided by cloud infrastructure system 1102. As part of the order, the customer may provide information identifying the application for which services are to be provided, and application storage profile information for that application.

[0081] 11, cloud infrastructure system 1102 may include an order management subsystem (OMS) 1120 configured to process new orders. As part of this processing, OMS 1120 may be configured to create an account for the customer if not already created, receive billing and / or account information from the customer to use for billing the customer for providing the requested services to the customer, verify the customer information, and, once verified, reserve the order for the customer and prepare the order for provisioning by coordinating various workflows.

[0082] Upon proper validation, the OMS 1120 may invoke an order provisioning subsystem (OPS) 1124 configured to provision resources for this order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring the resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for the order and the types of resources provisioned depend on the type of cloud service the customer has ordered. For example, following a workflow, OPS 1124 may be configured to determine the particular cloud service being requested and identify the number of pods that may be pre-configured for that particular cloud service. The number of pods allocated for an order may depend on the size / amount / level / scope of the service requested. For example, the number of pods to allocate may be determined based on the number of users the service is to support, the duration for which the service is requested, etc. The allocated pods may then be customized to the particular requesting customer to provide the requested service.

[0083] Cloud infrastructure system 1102 may send a response or notification 1144 to the requesting customer to indicate when the requested service will be available for use. In some examples, information (e.g., a link) may be sent to the customer that enables the customer to begin using and avail themselves of the benefits of the requested service.

[0084] Cloud infrastructure system 1102 may provide services to multiple customers. For each customer, cloud infrastructure system 1102 manages information related to one or more subscription orders received from the customer, maintains customer data related to the orders, and is responsible for providing the requested services to the customer. Cloud infrastructure system 1102 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected about the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and downtime. This usage information may be used to bill the customer. Billing may be on a monthly basis, for example.

[0085] Cloud infrastructure system 1102 may provide services to multiple customers in parallel. Cloud infrastructure system 1102 may store information about these customers, possibly including copyright information. In particular embodiments, cloud infrastructure system 1102 includes an identity management subsystem (IMS) 1128 configured to manage customer information and separate the managed information so that information about one customer is not accessible from information about another customer. IMS 1128 manages identity management. The system may be configured to provide various security-related services such as security services, information access management, authentication and authorization services, services for managing customer identities and roles and associated capabilities, etc.

[0086] 12 illustrates an exemplary computer system 1200 that may be used to implement certain embodiments. As shown in FIG. 12, computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with several other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. Storage subsystem 1218 may include non-transitory computer-readable storage media, including storage medium 1222 and system memory 710.

[0087] Bus subsystem 1202 provides a mechanism for allowing the various components and subsystems of computer system 1200 to communicate with each other as intended. While bus subsystem 1202 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1202 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus, etc., using any of a variety of bus architectures. For example, such architectures may include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the IEEE P1386.1 standard. These may include a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured according to specifications.

[0088] The processing subsystem 1204 controls the operation of the computer system 1200 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 1200 may be organized into one or more processing units 1232, 1234, etc. The processing units may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some embodiments, the processing subsystem 1204 includes one or more dedicated co-processors, such as graphics processors, digital signal processors (DSPs), etc. In some embodiments, some or all of the processing units of processing subsystem 1204 may use customized circuitry such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0089] In some embodiments, processing units within processing subsystem 1204 may execute instructions stored in system memory 1210 or computer-readable storage medium 1222. In various embodiments, the processing units may execute various program or code instructions and maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code to be executed may reside in system memory 1210 and / or computer-readable storage medium 1222, potentially including one or more storage devices. Through appropriate programming, processing subsystem 1204 may provide the various functions described above. In examples where computer system 1200 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0090] In particular embodiments, a processing acceleration unit 1206 may optionally be provided to accelerate the overall processing performed by the computer system 1200, to perform customized processing, or to offload portions of the processing performed by the processing subsystem 1204.

[0091] I / O subsystem 1208 can include devices and mechanisms for inputting information into computer system 1200 and / or devices and mechanisms for outputting information from or through computer system 1200. In general, use of the term "input device" is intended to include all conceivable types of devices and mechanisms for inputting information into computer system 1200. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices are devices that allow a user to control and interact with the input device. The user interface input devices may also include motion sensing and / or gesture recognition devices, such as a Microsoft Kinect® motion sensor, a Microsoft Xbox® 360 game controller, or a device that provides an interface for receiving input using gestures and voice commands. The user interface input devices may also include eye gesture recognition devices, such as a Google Glass® blink detector, that detects eye movements from the user (e.g., "blinks" while taking a picture and / or making a menu selection) and translates the eye gestures as input to the input device (e.g., Google Glass®). The user interface input devices may also include voice recognition sensing devices that allow the user to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0092] Other examples of user interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser range finders, and eye-tracking devices. User interface input devices may also include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0093] In general, the use of the term output device is intended to include all conceivable types of devices and mechanisms for outputting information from computer system 1200 to a user or to another computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be flat-panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / visual information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0094] The storage subsystem 1218 provides a repository or data store for storing information and data used by the computer system 1200. The storage subsystem 1218 provides a tangible, non-transitory, computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. Software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above may be stored in the storage subsystem 1218. The software may be executed by one or more processing units of the processing subsystem 1204. The storage subsystem 1218 may also provide a repository for storing data used in accordance with the teachings of the present disclosure.

[0095] The storage subsystem 1218 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in Figure 12, the storage subsystem 1218 includes a system memory 1210 and a computer-readable storage medium 1222. The system memory 1210 includes a volatile main random access memory (RAM) for storing instructions and data during program execution, and a fixed instruction memory (RAM) for storing instructions and data during program execution. The computer system 1200 may include several memories, including non-volatile read only memory (ROM) or flash memory for storing data. In some implementations, a basic input / output system (BIS) containing the basic routines that help transfer information between elements within the computer system 1200, such as during start-up. The system memory 1210 may typically be stored in ROM. RAM typically contains data and / or program modules currently being operated on and executed by the processing subsystem 1204. In some implementations, the system memory 1210 may be static random access memory (SRAM), dynamic random access memory (DRAM), or the like. It may include multiple different types of memory, such as dynamic random access memory (DRAM).

[0096] 12, system memory 1210 may load running application programs 1212, program data 1214, and operating system 1216, which may include various applications such as a web browser, middle-tier applications, relational database management systems (RDBMS), etc. By way of example, operating system 1216 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS®, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, etc.

[0097] The computer-readable storage medium 1222 may store programming and data constructs that provide functionality of some embodiments. The computer-readable storage medium 1222 may provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1200. Software (programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above may be stored in the storage subsystem 1218. By way of example, the computer-readable storage medium 1222 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD-ROM, a DVD, an optical disk drive such as a Blu-Ray® disk, or other optical media. The computer-readable storage medium 1222 may include, but is not limited to, a Zip® drive, a flash memory card, a universal serial bus (USB) flash drive, a secure digital (SD) card, a DVD disk, a digital video tape, etc. The computer-readable storage medium 1222 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROM, etc., SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0098] In particular embodiments, storage subsystem 1218 may also include a computer-readable storage medium reader 1220 that may further connect to a computer-readable storage medium 1222. Reader 1220 may include a memory device such as a disk, flash drive, etc. The device may be configured to receive and read data from the device.

[0099] In particular embodiments, computer system 1200 may support virtualization techniques, including, but not limited to, virtualization of processing and memory resources. For example, computer system 1200 may provide support for running one or more virtual machines. In particular embodiments, computer system 1200 may execute a program such as a hypervisor that facilitates configuration and management of virtual machines. Each virtual machine may be assigned memory, computing (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs independently from other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating systems run by other virtual machines executed by computer system 1200. Thus, potentially multiple operating systems may be running simultaneously by computer system 1200.

[0100] The communications subsystem 1224 provides an interface to other computer systems and networks. The communications subsystem 1224 serves as an interface for sending and receiving data between other systems and the computer system 1200. For example, the communications subsystem 1224 may enable the computer system 1200 to establish a communications channel over the Internet to one or more client devices to send and receive information to and from the one or more client devices.

[0101] The communications subsystem 1224 may support both wired and / or wireless communications protocols. For example, in one embodiment, the communications subsystem 1224 includes radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), WiFi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof). Global Positioning System (GPS) receiver In some embodiments, the communications subsystem 1224 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0102] The communications subsystem 1224 may receive and transmit data in various formats. For example, in some embodiments, the communications subsystem 1224 may receive incoming communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., among other formats. For example, the communications subsystem 1224 may be configured to receive (or transmit) data feeds 1226 in real time from users of social media networks and / or other communications services, such as web feeds such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0103] In particular embodiments, the communications subsystem 1224 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1228 of real-time events and / or event updates 1230, which may be continuous or infinite in nature without a clear end. Examples of applications that generate continuous data include, for example, sensor data applications, financial stock tickers, network performance measurement tools (e.g., network monitoring applications), and the like. and traffic management applications), clickstream analysis tools, and automobile traffic monitoring.

[0104] Communications subsystem 1224 may be configured to communicate data from computer system 1200 to other computer systems or networks. This data may be communicated in a variety of different formats, such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., to one or more databases that may be in communication with one or more streaming data source computers coupled to computer system 1200.

[0105] Computer system 1200 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® cellular phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or other data processing system. Due to the constantly changing nature of computers and networks, the description of computer system 1200 shown in FIG. 12 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in FIG. 12. Those skilled in the art will recognize other ways and / or methods for implementing various embodiments based on the disclosure and teachings herein.

[0106] FIG. 13 illustrates an exemplary data flow through an architecture 1300 and system for creating metric indicator recommendations based on identifying and selecting the most informational content for a user. The most informational content may be information about metric indicators that have any statistically significant changes associated with them or any statistically significant anomalies associated with them over a period of time. For example, a KPI may be identified as an anomaly if its trend or trend values ​​over a particular time period are inconsistent with the distribution of expected values ​​based on a forecast by a multivariate time series model. Providing this information to a user may be beneficial regardless of whether the user has requested that metric indicator. Furthermore, a user may be limited to the specific metric indicators to which they have access. For any given user, relevant metric indicators to which the user has access are considered. For example, if a total revenue KPI has an anomaly signal associated with it, but the user does not have access to the total revenue KPI, the total revenue KPI will not be provided to the user. Furthermore, while a user typically views headcount KPIs and attrition KPIs, if there are no anomalies or other signals associated with those KPIs, those KPIs are not always removed from the user's view. Alternatively, user behavior may be added to the signal detection to provide additional KPIs for the user to view, so that the user can see not only KPIs that the user would self-select, but also KPIs that have information that is valuable to the user.

[0107] Architecture 1300 includes databases 1302, 1304, and 1306. Databases 1302, 1304, and 1306 can be any suitable external data sources to provide enterprise data to the smart analytics system. Database 1302 is an external data source containing data from the company, including customer data, sales data, support data, etc. Database 1306 is data from the public and other external sources, such as data from the Department of Labor. Data warehouse 1304 is a data source directly accessed by components included in the system. The external data from data sources 1302 and 1306 is used to populate data source 1304 for use by the system.

[0108] Data from the data warehouse 1304 is used in the KPI selector 1310 to identify signal anomalies. The KPI selector 1310 can identify metric indicators that may indicate anomalous signals, for example, because they may be trending outside of thresholds. To identify metric indicators, for a given metric indicator, the KPI modeler and predictor 1312 can generate a multivariate dynamic dependency model. The multivariate dynamic dependency model can be a VARIMAX (Vector Auto Regressive Integrated Moving Average with Exogenous Variables) model. The KPI selector 1310 may be configured to perform a KPI selection process, including but not limited to, multivariate variable attention temporal attention long short-term memory (MIM) using time series data for the metric indicator. A prediction of a first set of values ​​for a first variable or attribute of the metric indicator is generated based on a dynamic dependency model. Values ​​from the model may be compared to the actual values ​​to find statistical deviations using a divergence test to determine whether the distribution of observed values ​​significantly diverges from the forecasted predictions. Divergence tests include, but are not limited to, the Kullback-Leibler Information Test. If the statistical deviation is significant (e.g., greater than two standard deviations away from the center of gravity of the distribution), the KPI selector 1310 identifies the metric indicator as being relevant or as having some significant information indicating that a person should look at it. Therefore, the KPI selector 1310 selects the KPI. If there are no anomalies with the metric indicator, the KPI is not selected for highlighting to the user. Alternatively, previously forecasted values ​​of the metric indicator may be identified and compared with the current metric indicator value. An anomaly is identified when a metric indicator value deviates statistically (e.g., by two standard deviations) from a forecast value across multiple time points. If the metric indicator model suggests a problem or anomaly, the metric indicator may be selected for inclusion in a KPI suite for the user.

[0109] In some embodiments, selected KPIs (metric indicators) may be provided to the derivative integrator 1318 without further analysis. However, in some embodiments, the KPI explainer 1314 may generate an explanation for the anomaly based on a model. For example, a model of the metric indicator, the current metric indicator value, and the values ​​of the contributing variables at that time may be used to identify attributes that are significant to the metric indicator at the relevant time point. Derivatives of the attributes may be calculated in the model, and the most influential ones are identified because they will have the largest derivative values ​​based on computational techniques, including, but not limited to, locally interpretable model explanations and Shapley additive explanations. The explanations serve as locally faithful linear models that act as surrogates for the original nonlinear model. Metric indicators may also be analyzed using machine learning models to determine which metric indicators have statistically significant changes over the time period of interest. Percent changes or z-scores of metric indicators may be calculated to determine the practical significance of the changes. In some embodiments, explanations are not generated, for example, because the system is unable to converge to a poor explanation or there is insufficient data to calculate such an explanation.

[0110] Once the explanation is generated, the KPI simulator 1316 can generate recommendations for remediating the signal anomaly. The recommendations can be generated by running what-if simulations of the model by changing the identified attributes with the greatest impact to reach the most or more desirable or closest non-anomalous value of the KPI before the anomaly occurs. For example, what-if simulations can be used to measure the metric indicator values ​​in a model. The most influential attributes can be modified until the metric indicator value matches the predicted metric indicator value. Once the attribute modifications values ​​are used, recommendations can be generated. The signal anomaly, the modeled signal anomaly, the explanation, and the recommendation can be provided to a deriving integrator 1318.

[0111] The metadata indexing and tagging engine 1308 can receive data from the data warehouse 1304 and tag or apply metadata to the data, which is then sent to the metadata index database 1330 .

[0112] Using the graphical user interface represented by browser analytics 1342, a user generates activities from creating, using, and viewing various metric indicators. Each action is recorded by action recorder 1326. The recorded actions are analyzed to extract activity data and context data. The activity data is stored in activity working memory 1328, and the context data is stored in context working memory 1324. As an example, if a user logs in and immediately views a headcount metric indicator and then a attrition metric indicator, the activity may be that the KPIs are viewed sequentially for a certain length of time, etc. The context data may be, for example, the time, device, meetings attended, etc., at which the user viewed the two metrics. The activity data and context data are combined, for example, for all related actions performed over a period of time to generate episode memory 1322. The episode memory 1322 is clustered into similar episodes by clustering engine 1320. Episode information and clusters related to context and data are stored in relationship data store 1332, role and KPI data store 1334, and action knowledge data store 1336. User-specific usage data stored in data stores 1332, 1334, and 1336 is fed to inductive integrator 1318. Over time, individual user episodes can reveal patterns, such as a user always logging in and immediately viewing headcount metrics, then attrition metrics. Furthermore, over time, clustered data about many users can be captured and used to generate personalized recommendations for the user based on what many other users do, in what order they view KPIs, etc. All of this information can be provided to the inductive integrator.

[0113] The induction integrator 1318 generates metric recommendations using signal anomaly information from the KPI selector 1310 and the KPI modeler and predictor 1312. In some embodiments, user-specific information is combined to generate metric recommendations. For example, if a revenue metric is identified by the KPI selector 1310, the user has access to the revenue metric, and the user typically reviews the profit metric. Even if a profit metric was not captured by the KPI selector 1310, the user may be provided with a set including the revenue metric and the profit metric. Metric recommendations and other information are provided to the presentation manager 1338. Metric recommendations may be provided to the user in the related information sidebar 1340, which may be, for example, recommendations 315 of FIG. 3. The related information sidebar 1340 may be, for example, the related metric indicators 315 shown in FIG. 3. Analytics information (used to generate the metric indicator cards and sets described above) is presented to the user via a graphical user interface. Browser analytics 1342 are presented to the user using a browser interface. Mobile analytics 1344 are presented to the user using the mobile application. The data (analytics) contained in the metric indicators is the same whether the interface is a browser or a mobile device.

[0114] In order to display the metric information in the browser-based analytics and / or mobile analytics 144 user interface, a graphical representation of the KPI must be generated, i.e., a KPI card must be generated. For each KPI in the system, A PI modeler and predictor 1312 is used to generate a model of the metric indicator (e.g., a random forest or XGBoost model). Attributes are included in the model as independent variables. Using the model, the attributes with the highest entropy change are used to evaluate, for example, Shapley's additive explanatory value or locally interpretable models. Calculate local interpretable model-agnostic explanations (LIME) values When generating a graphical representation of a metric indicator, a card is generated for display, such as the card shown in the example user interface herein, including card 505 of FIG. 5. The card typically includes some information associated with the attribute with the highest (largest) entropy change identified above. In this way, the most relevant content associated with the metric indicator is presented for viewing by the user.

[0115] Figure 14 shows a method 1400 that may be performed using a system having architecture 1300 as described in Figure 13. Method 1400 may begin at step 1405, where the system receives first data associated with an activity performed by a user in a graphical user interface. For example, action recorder 1326 may receive and record activity data of the user's interaction with metrics using browser-based analytics 1342 and / or mobile analytics 1344.

[0116] In step 1410, activity data and context data may be extracted from the first data. For example, context data may be extracted and stored in context working memory 1324, and activity data may be extracted and stored in activity working memory 1328. Activity information is obtained directly from user interactions with user interface elements in the form of mouse clicks, scrolling, and typing in relation to KPIs, cards, and sets, and the sequence of these actions is stored as a temporal activity graph. Similarly, context regarding each of the above actions, its predecessor actions, resulting actions, the user's roles and responsibilities, the user's collaborative activities with other users, and the user's sharing and email activity regarding KPIs is also attached as a graph to each node in the above activity graph and stored.

[0117] In step 1415, episode data is generated from the activity data and the context data for all associated activities performed by the user. Specifically, if a user sees several associated metric indicators, the view may be recognized as an episode and stored accordingly.

[0118] At step 1420, second data of metric indicators indicative of an anomalous signal is received. For example, the signal anomaly selector 1310 can receive information from the data warehouse 1304 regarding metric indicators that, when analyzed, reveal anomalies.

[0119] In step 1425, the second data is modeled by the KPI modeler and forecaster 1312 to identify anomalous signals as described above. For example, the modeler can generate a dynamic dependency model using the time series data of the metric indicator. A prediction of a first set of values ​​for a first variable or attribute of the metric indicator is generated based on the dynamic dependency model. Values ​​from the model may be compared to the actual values ​​to generate a statistical deviation using a divergence test. If the statistical deviation is significant (e.g., two standard deviations above), the KPI selector 1310 identifies the metric indicator as being relevant or as having some significant information that indicates that a person should look at it. This deviation indicates an anomaly, and thus a signal anomaly is identified and verified.

[0120] In step 1430, an explanation of the abnormal signal based on the modeled abnormal signal may be generated. Such an explanation may include, for example, factors that led the metric indicator to trend out of a threshold. The explanation may be found by finding the derivatives of the model's attributes using a model of the metric indicator and the current metric indicator value, where the attributes are each represented as independent variables in the model. The attribute with the highest partial derivative is identified as being most influential at the time the metric indicator value is at that point. These attributes are identified to explain the anomaly using techniques such as Shapley's additive explanations or LIME.

[0121] In step 1435, metric indicator recommendations based on the episode data and the modeled anomaly signal are generated. The recommendations may include steps that can be taken to remediate the anomaly. Specifically, the KPI path directive 1316 may generate the recommendations. What-if simulations of the model may be performed to determine the most desirable state for the metric indicator value. What-if simulations are performed by changing the values ​​of the identified attributes that have the most impact on the performance of the business. For example, if a metric indicator deviates from a forecast value, what-if simulations can be performed to change the forecast. One can attempt to modify the attribute to obtain a metric value that corresponds to a value or values ​​within the expected distribution.

[0122] In step 1440, a metric indicator recommendation including an explanation and recommendations for addressing the abnormal signal is generated and displayed in the graphical user interface. The metric recommendation may be displayed, for example, in a sidebar while the user is viewing the selected metric indicator. For example, the metric indicator recommended for viewing may be generated as a card with several values ​​or information, such as a graph or values ​​showing the metric indicator value over time, in addition to information about the most relevant attributes of the metric indicator identified by the KPI explanation portion 1314 and the KPI path indication portion 1316. The user interfaces herein provide examples of KPI cards and other information that can be generated for user viewing.

[0123] In step 1445, metric indicator recommendations, including explanations and recommendations for addressing the abnormal signal, are displayed within the graphical user interface. For example, the visual depiction may be provided within the associated information sidebar 1340 or more directly on the user's screen. As an example, the information box 210 in FIG. 2A provides some notification information that new employment attrition is at 12%, and one explanation provided is that manufacturing attrition is four times higher than the expected rate. This type of information can help the user understand why values ​​are the way they are.

[0124] FIG. 15 illustrates an example method 1500 for providing high-value metric indicators to a user. Method 1500 may be performed, for example, by smart analytics system 100 and / or smart analytics architecture 1300. Method 1500 begins in step 1505, where KPI modeler and predictor 1312 generates a model for each metric indicator, with each attribute as an independent variable. A model is created for each KPI to identify which attributes are most relevant to the KPI, at least over the time period of interest. In this manner, a KPI card may be generated for any KPI that may be presented to a user. Note that for a given user, some KPIs may not be available, and thus, in some embodiments, KPIs that are not accessible to the user may not be analyzed in step 1505.

[0125] In step 1510, the KPI selector 1310 may select at least one attribute with the largest change contribution based on the model for each metric indicator. For example, the model generated by KPI modeler 1312 may be used to take the derivative of each attribute and find the one with the largest derivative (change contribution). The attribute with the highest change contribution is the most significant or influential to that metric indicator.

[0126] In step 1515, the KPI selector 1310 can generate a set of metric indicators based on selecting a subset of metric indicators, where the subset is selected based on selecting metrics with the largest statistically significant change. The metric with the largest entropy change can be determined by identifying whether the metric indicator may have an anomaly associated with it. As described above, one method for determining whether a metric indicator has an anomaly includes generating a dynamic dependency model using the time series data of the metric indicator. A first set of predictions for values ​​of a first variable or attribute of the metric indicator is generated based on the dynamic dependency model. Values ​​from the model may be compared with actual values ​​to generate a statistical deviation using a statistical divergence test. Statistical divergence tests include, but are not limited to, the Kullback-Leibler Information Test. If the statistical deviation is significant (e.g., equivalent to two or more standard deviations), the KPI selector 1310 identifies the metric indicator as being relevant or as having some statistically significant information indicating that a person should look at it. This deviation indicates an anomaly, and the metric indicator with the anomaly is included in the set of metric indicators. In some embodiments, user personalization may also be used to add metric indicators to the set. For example, if information in important KPIs and relationships database 1332 includes information that a particular user prefers certain metric indicators to view regularly, those KPIs may also be included in the set. Important KPIs and relationships database 1332 may obtain this user-specific information from clustering engine 1320.

[0127] In step 1520, the presentation manager can generate a graphical representation of each of the subset of metric indicators, each graphical representation including at least one attribute with the largest entropy change. Stated another way, the metric card can include information about the attributes that most affect the metrics identified by the KPI selector 1310 when each KPI is analyzed in 1510.

[0128] In some embodiments, the metric indicators are ordered based on statistical significance and / or practical significance. The ordering is based on the z-score (normalized standard deviation) distance between the observed distribution and the expected forecasted prediction, with the ranking in descending order. Practical significance is determined based on the product of how infrequently users within the enterprise probe anomalies in that area and how large the deviation is in terms of the z-score. Once the metric indicators are ordered and the metric indicators with the highest informational value are selected (subset), a graphical display of each of the ordered subset of metric indicators may be generated.

[0129] In step 1525, a graphical representation of the metric indicators, which may be ordered, is provided to a user's display for viewing.

[0130] While specific embodiments have been described, various modifications, variations, alternative constructions, and equivalents are possible. The embodiments are not limited to operation in any particular data processing environment, but operate freely in multiple data processing environments. Additionally, while certain embodiments have been described using a particular sequence of transactions and steps, those skilled in the art will recognize that this is not limiting. While some flowcharts describe operations as sequential processes, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. The process may have additional steps not included in the figures. Various features and aspects of the above-described embodiments may be used individually or together.

[0131] Additionally, while particular embodiments have been described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are possible. Particular embodiments may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or any combination of different processors.

[0132] When a device, system, component, or module is described as being configured to perform a certain operation or function, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by executing computer instructions or code, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication; different pairs of processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0133] Specific details are provided in this disclosure to facilitate a thorough understanding of the embodiments. However, the embodiments may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the embodiments. This description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of other embodiments. Rather, the foregoing description of the embodiments will provide one skilled in the art with an enabling description for implementing various embodiments. Various changes may be made in the function and arrangement of elements.

[0134] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and changes may be made without departing from the broader spirit and scope of the appended claims. Accordingly, while particular embodiments have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

Claims

1. 1. A computer-implemented method comprising: For each metric indicator of a company's multiple metric indicators, training a model of said metric indicator using a machine learning process, said model having each attribute of a plurality of attributes explaining changes in said metric indicator as an independent variable of said model, and configured to output a value of said metric indicator corresponding to said plurality of attributes and a rate of change of said value, and to calculate a rate of change for each of said plurality of attributes, said method further comprising: for each metric indicator of said plurality of metric indicators of said company: and selecting, based on the model, at least one attribute of the plurality of attributes having a greatest rate of change using the model for the metric indicator, wherein the greatest rate of change is based on the rate of change calculated by the model, the method further comprising, for each metric indicator of the plurality of metric indicators for the company: Obtaining the previous expected value of the current metric indicator; selecting the metric indicator as an element of the subset of the plurality of metric indicators if the previous expected value differs from the current value of the metric indicator; generating a graphical user interface for each of the subset of the plurality of metric indicators, wherein each graphical user interface of the subset of the plurality of metric indicators includes a value of a metric indicator corresponding to the at least one attribute of the plurality of attributes, the method further comprising, for each metric indicator of the plurality of metric indicators for the company: providing the graphical user interface to a device of the user for display to the user, a computer-implemented method.

2. The method further comprises:

2. The computer-implemented method of claim 1, comprising identifying the at least one attribute when the previous expected values ​​for the subset of the plurality of metric indicators differ from current values ​​of the metric indicators by applying the current values ​​of the metric indicators to the model; calculating a rate of change for each of the plurality of attributes of the model; and selecting the subset of each of the plurality of attributes based on the calculated rate of change.

3. The method further comprises:

3. The computer-implemented method of claim 2, further comprising changing the value of the metric indicator to the expected value if the metric indicator deviates from the expected value.

4. The method further comprises: recording said user's input to a user interface; generating a usage pattern of the user for at least one metric indicator based on the recorded input; generating a set of metric indicators including the at least one metric indicator based on selecting the subset; The computer-implemented method of any one of claims 1 to 3, wherein the input includes at least one of a mouse click, scrolling, and typing.

5. generating a graphical user interface for each of the subset of the plurality of metric indicators, The computer-implemented method of any one of claims 1 to 4, comprising creating a metric card including values ​​of metric indicators corresponding to the at least one attribute.

6. The method further comprises: identifying a second subset of the plurality of metric indicators to which the user has access; and selecting the subset of the plurality of metric indicators from the second subset of the plurality of metric indicators.

7. The method further comprises: tracking usage patterns of a plurality of users for the user interface; identifying time series data of the number of users viewing each of the plurality of metric indicators based on the usage patterns of the plurality of users; generating a graphical trend card including the time series data; and providing the graphical trend card to one or more of the plurality of users in the user interface.

8. 1. A system comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations including: For each metric indicator of a company's multiple metric indicators, training a model of the metric indicator using a machine learning process, the model having each attribute of a plurality of attributes explaining changes in the metric indicator as an independent variable of the model, and configured to output a value of the metric indicator corresponding to the plurality of attributes and a rate of change of the value, and to calculate a rate of change for each of the plurality of attributes, the operations further comprising: for each metric indicator of the plurality of metric indicators for the company: and selecting, based on the model, at least one attribute of the plurality of attributes having a greatest rate of change using the model for the metric indicator, the greatest rate of change being based on the rate of change calculated by the model, and the operations further include, for each metric indicator of the plurality of metric indicators for the company: Obtaining the previous expected value of the current metric indicator; selecting the metric indicator as an element of the subset of the plurality of metric indicators if the previous expected value differs from the current value of the metric indicator; generating a graphical user interface for each of the subset of the plurality of metric indicators, wherein each graphical user interface for the subset of the plurality of metric indicators includes a value of a metric indicator corresponding to the at least one attribute of the plurality of attributes, and the operations further include, for each metric indicator of the plurality of metric indicators for the company: providing the graphical user interface to a device of the user for display to the user.

9. The operation further comprises:

10. The system of claim 8, further comprising generating a multivariate time series model that predicts an attribute value for each of the plurality of metric indicators from the plurality of metric indicators using time series data for each of the plurality of metric indicators.

10. The operation further comprises: The system of claim 9 , further comprising generating a forecast of attribute values ​​for each of the plurality of metric indicators using the multivariate time series model.

11. The operation further comprises: comparing the predictions of the attribute values ​​with actual attribute values ​​to generate statistical deviations using divergence tests for the actual attribute values; The system of claim 10 , wherein the divergence test determines whether the distribution of actual attribute values ​​diverges significantly from the prediction.

12. Selecting the subset of the plurality of metric indicators comprises: The system of claim 11 , further comprising identifying the subset of the plurality of metric indicators as having a significant statistical deviation between the prediction of the attribute value and the actual attribute value.

13. Selecting the subset of the plurality of metric indicators from the plurality of metric indicators further comprises: The system of claim 12 , comprising selecting the subset of the plurality of metric indicators based on metadata including user-personalized specific information for previously regularly viewed metric indicators.

14. 1. A computer program that, when executed by one or more processors, causes the one or more processors to perform operations, said operations comprising: For each metric indicator of a company's multiple metric indicators, training a model of the metric indicator using a machine learning process, the model having each attribute of a plurality of attributes explaining changes in the metric indicator as an independent variable of the model, and configured to output a value of the metric indicator corresponding to the plurality of attributes and a rate of change of the value, and to calculate a rate of change for each of the plurality of attributes, the operations further comprising: for each metric indicator of the plurality of metric indicators for the company: and selecting, based on the model, at least one attribute of the plurality of attributes having a greatest rate of change using the model for the metric indicator, the greatest rate of change being based on the rate of change calculated by the model, and the operations further include, for each metric indicator of the plurality of metric indicators for the company: Obtaining the previous expected value of the current metric indicator; selecting the metric indicator as an element of the subset of the plurality of metric indicators if the previous expected value differs from the current value of the metric indicator; generating a graphical user interface for each of the subset of the plurality of metric indicators, wherein each graphical user interface of the subset of the plurality of metric indicators includes a value of a metric indicator corresponding to the at least one attribute of the plurality of attributes, and the operations further include, for each metric indicator of the plurality of metric indicators for the company: providing the graphical user interface to a device of the user for display to the user.

15. The computer program product of claim 14 , wherein the subset of the plurality of metric indicators comprises a collection of graphical cards providing data associated with the subset of metric indicators.

16. The computer program product of claim 15 , wherein the graphical user interface for each of the subset of the plurality of metric indicators comprises a graphical dashboard that displays a collection of the graphical cards.

17. 17. The computer program product of claim 15 or 16, wherein the collection of graphical cards displays graphs of the data associated with the subset of metric indicators, and the type of graph is determined automatically based on metadata including user-specific information personalized for the user.

18. A computer program according to any one of claims 15 to 17, wherein the collection of graphical cards displays actual and predicted attribute values.

19. 19. The computer program of claim 14, wherein training data for the model of the metric indicator is selected from data stored in a data warehouse for the enterprise.

Images