Application Health Monitoring

Abstract

To provide a system, method and program for monitoring the application health.SOLUTION: In a platform for an application health monitor service 111, a communication channel with an agent installed on each client 101 is activated and, while the communication channel is active, usage data is collected via the agent, where the usage data characterizes usage regarding a plurality of different applications accessed via the client. The usage data is analyzed to determine metrics associated with the plurality of different applications. An interactive user interface dashboard providing the determined metrics is provided.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] An organization collects usage information about software applications deployed within the organization. The collected usage information may include application performance and availability metrics for different types of applications, including desktop applications and software-as-a-service (SaaS) applications. Typically, the usage information is provided by the application vendor and / or collected through user feedback and user surveys. The organization may then rely on the information when evaluating the costs and benefits of the application to the organization. For example, the usage information may be a factor in deciding whether to continue purchasing the application, whether to make the application more widely available to additional users within the organization, or whether to switch to a different application. [Brief explanation of the drawings]

[0002] Various embodiments of the present invention are disclosed in the following detailed description and the accompanying drawings.

[0003] [Figure 1] FIG. 1 is a block diagram illustrating one embodiment of a platform for application health monitoring services.

[0004] [Figure 2] 1 is a block diagram illustrating an embodiment of a client device configured for application monitoring.

[0005] [Figure 3] FIG. 1 is a block diagram illustrating one embodiment of an application health monitoring service.

[0006] [Figure 4] 10 is a flow chart illustrating one embodiment of a process for monitoring the health of a client application.

[0007] [Figure 5] 10 is a flow chart illustrating one embodiment of a process for configuring a client for application health monitoring.

[0008] [Figure 6] 10 is a flow chart illustrating one embodiment of a process for analyzing collected application health data.

[0009] [Figure 7] 1 is a flow chart illustrating one embodiment of a process for providing application health results.

[0010] [Figure 8] 10 is a flow chart illustrating one embodiment of a process for identifying and returning unnecessary application licenses.

[0011] [Figure 9] FIG. 1 is a functional diagram illustrating a computer system programmed to monitor application health.

[0012] [Figure 10] FIG. 1 illustrates one embodiment of a user interface for viewing analyzed usage metrics for multiple applications via an interactive user interface dashboard.

[0013] [Figure 11] FIG. 1 illustrates one embodiment of a user interface for viewing analyzed usage metrics collected from multiple users for a particular application. DETAILED DESCRIPTION OF THE INVENTION

[0014] The present invention may be embodied in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. These embodiments, or any other form the present invention may take, may be referred to herein as technology. In general, the order of steps in a disclosed process may be varied within the scope of the present invention. Unless otherwise noted, components, such as a processor or memory, described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term “processor” refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0015] The following is a detailed description of one or more embodiments of the present invention with reference to figures that illustrate the principles of the invention. While the present invention has been described in connection with such embodiments, it is not limited to any particular embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters that are well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0016] Application health monitoring is disclosed. For example, an application health monitoring service platform monitors client application usage and provides metrics related to application health via a cloud-based application health monitoring service. In some embodiments, a client device is configured with an agent for monitoring applications and collecting usage data (such as application uptime, crashes, activity, resource usage, and user usage, among other usage data). Cloud-based applications, including software as a service (SaaS), may be monitored using the configured agent and / or via a configured browser extension. For example, a browser extension may be installed on the client device to monitor and collect usage data for web-based applications. In various embodiments, the monitored applications are based on an inclusion list configured by an information technology team managing the device. The collected usage data is provided by the agent installed on the client device to an application health monitoring service, which may perform analysis of the data, such as aggregating and auditing the usage data, to provide corresponding usage metrics. In various embodiments, the determined metrics are provided for review via an interactive user interface dashboard. For example, an IT operations team may access the determined metrics via a web-based interface of a cloud-based application health monitoring service. In various embodiments, among other metrics, the analyzed metrics may include audit data regarding license usage, allowing an organization to optimize paid license purchases and usage. Similarly, the analyzed metrics may include uptime metrics, allowing an organization to validate guaranteed uptime agreements provided by application vendors.In some embodiments, the application health monitoring service provides audits and / or license usage (e.g., refund requests and purchase requests) to a software asset management (SAM) platform, for example, to initiate refunds for unused licenses or to purchase additional licenses.

[0017] In some embodiments, a communication channel is activated with an agent installed on the client. For example, a communication channel is activated between an application health monitoring service and an agent installed and running on the client device. The client device may be a computing device used to run applications, including both desktop applications and cloud or software-as-a-service (SaaS) applications. In some embodiments, a browser extension is installed on the client device and configured to monitor cloud-based applications accessed via a web browser, and the agent installed on the client device is configured to monitor desktop and internet applications. In some embodiments, while the communication channel is active, usage data is collected via the agent, where the usage data characterizes usage for multiple different applications accessed via the client. For example, the installed agent and / or browser extension monitors and collects usage data for different applications accessed via the client. The collected usage data is provided by the agent via the established communication channel. For example, the usage data is provided from the agent to the application health monitoring service.

[0018] In some embodiments, the usage data is analyzed to determine metrics associated with multiple different applications. For example, the application health monitoring service analyzes received usage data for multiple different applications accessed via clients. In some embodiments, an interactive user interface dashboard presenting the determined metrics is provided. For example, a web-based health monitoring application is provided by the application health monitoring service for viewing and interacting with the determined metrics associated with the multiple different applications. In various embodiments, the analyzed usage data and associated metrics may include usage data for multiple different applications across multiple clients. For example, the application health monitoring service may receive and analyze usage data from various clients configured for application monitoring. The analysis performed by the application health monitoring service may include determining metrics associated with auditing licensed application usage, such as auditing and providing reports regarding application uptime and / or usage patterns of licenses associated with the monitored applications.

[0019] 1 is a block diagram illustrating one embodiment of a platform for an application health monitoring service. In the illustrated example, clients 101, 103, and 105 are network clients configured with application health monitoring agents. Clients 101, 103, and 105 are communicatively coupled to application health monitoring service 111 via network 151. Network 151 may be a public network or a private network. In some embodiments, network 151 is a public network such as the Internet. Application health monitoring service 111 provides a cloud-based service for investigating usage metrics associated with clients 101, 103, and 105. For example, in some embodiments, application health monitoring service 111 provides an interactive user interface dashboard for investigating application usage metrics.

[0020] In some embodiments, clients 101, 103, and 105 are each network client devices for running applications (e.g., desktop applications, cloud applications, including Software as a Service (SaaS) applications, or web-based applications). Each of clients 101, 103, and 105 is configured with an application health monitoring agent (not shown) that collects and provides usage data for the monitored applications to application health monitoring service 111. For example, a communication channel may be established between application health monitoring service 111 and each of clients 101, 103, and 105 to provide the collected usage data. In some embodiments, a communication channel is established between application health monitoring service 111 and a corresponding application health monitoring agent installed on each of clients 101, 103, and 105.

[0021] In some embodiments, each of clients 101, 103, and 105 is configured with a browser extension or plug-in (not shown) for monitoring web-based applications accessed via a web browser. The configured browser extension can monitor and collect usage data associated with applications accessed via the associated web browser. For example, SaaS application usage data can be collected by the browser extension when the SaaS application is accessed via the associated web browser. In various embodiments, the collected usage data is provided by the browser extension to an application health monitoring agent on the same client device. The application health monitoring agent then provides the collected usage data to the application health monitoring service 111. In some embodiments, the application health monitoring agent is implemented as a browser extension. In some embodiments, the application health monitoring agent is implemented as a standalone application (e.g., an agent daemon or service) running on the client device, and the agent communicates with the browser extension to collect and forward browser-based application usage data to the application health monitoring service 111. Although described as an application health monitoring agent, in various embodiments, the actual implementation of the agent may be a general-purpose monitoring agent capable of monitoring application health, activity, features, functions, capabilities, data, etc., among other metrics.

[0022] In some embodiments, the services provided by the application health monitoring service 111 are accessed by users of network clients (e.g., clients 101, 103, and 105). For example, an IT administrator may use the network clients (e.g., clients 101, 103, and 105) to configure the clients for application monitoring. As another example, an IT administrator may use the network clients (e.g., clients 101, 103, and 105) to access a dashboard to review and interact with application usage metrics provided by the application health monitoring service 111. In some embodiments, the dashboard is accessed via a web browser, and an interactive user interface dashboard is provided, at least in part, by the application health monitoring service 111. In some embodiments, the dashboard is accessible via a custom application based on usage metrics provided by the application health monitoring service 111. For example, the custom application may include a system tray application, a chatbot application, and / or a cloud-based desktop application, among others, to interact with usage metrics provided by the application health monitoring service 111.

[0023] In some embodiments, the application health monitoring service 111 is a cloud server that provides application health monitoring services. The provided application health monitoring services may include the ability to configure and manage application monitoring on clients (e.g., clients 101, 103, and 105) and the ability to receive and analyze collected application usage data. For example, the application health monitoring service 111 may collect application usage data from clients 101, 103, and 105 for analysis. The analyzed usage data may be provided as usage metrics via an interactive user interface dashboard. In some embodiments, the analysis is provided to one or more software asset management (SAM) platforms to initiate action based on identified application usage (e.g., insufficient usage of identified paid licenses or application uptime metrics that do not meet application uptime guarantees). In various embodiments, the application usage metrics may include aggregate results, such as daily, weekly, monthly, and / or other results based on differently configured time frames, as well as location-dependent results, such as results for a specific region and / or computer network location.

[0024] Although some components are shown only one at a time to simplify the diagram of FIG. 1 , any of the components shown in FIG. 1 may be present in additional ways. For example, application health monitoring service 111 may include one or more cloud servers and one or more databases utilized by the cloud servers. Furthermore, clients 101, 103, and 105 are examples of client devices for client application monitoring and / or management of application monitoring. In some scenarios, each of clients 101, 103, and 105 may function only as a client for client application monitoring or as a client for management of application monitoring (via application health monitoring service 111), but not both. Although three clients (clients 101, 103, and 105) are shown, many additional clients may exist and communicate with and / or be managed from application health monitoring service 111. In some embodiments, components not shown in FIG. 1 may be present.

[0025] FIG. 2 is a block diagram illustrating one embodiment of a client device configured for application monitoring. In the illustrated example, client 201 is a network client and includes browser 211 and application health monitoring agent 251. Browser 211 and application health monitoring agent 251 establish network connections 213 and 253, respectively. In various embodiments, application health monitoring agent 251 is an application (e.g., an operating system application) and is configured as an agent for monitoring client 201. While configured with the ability to monitor the application health of client 201, in various embodiments, an actual implementation of application health monitoring agent 251 may be configured as a general-purpose monitoring agent with the ability to monitor application health, activity, features, functions, capabilities, data, etc., among other metrics. Network connection 213 is used by browser 211 to access browser-based applications (e.g., cloud applications or web-based applications, including software as a service (SaaS) applications). Browser 211 includes browser monitoring extension 221 for monitoring usage of application access via browser 211. Browser monitoring extension 221 is communicatively coupled to application health monitoring agent 251, for example, to provide collected browser usage data. In addition to monitoring client 201, application health monitoring agent 251 provides collected usage (e.g., usage data collected via browser monitoring extension 221) to a cloud-based application health monitoring service using network connection 253. In various embodiments, application health monitoring of client 201, including configuration of browser monitoring extension 221 and / or application health monitoring agent 251, may be configured remotely via network connections 213 and 253.In some embodiments, the cloud-based application health monitoring service is application health monitoring service 111 of FIG. 1, and client 201 is client 101, 103, and / or 105 of FIG.

[0026] In some embodiments, client 201 is a network client, such as a desktop computer, laptop, mobile device, tablet, kiosk, voice assistant, wearable device, or another networked computing device. Client 201 may be managed by an IT operations group and configured to access applications (e.g., applications approved by the IT operations group). A user of client 201 may utilize client 201 to execute and operate applications, including local applications and / or web-based applications. For example, applications utilized by a user of client 201 may include applications installed on client 201 and applications accessed via browser 211. Client 201 is configured for application health monitoring by installing application health monitoring agent 251 and / or browser monitoring extension 221 on client 201. In some embodiments, client 201 may be configured for application health monitoring via application health monitoring agent 251 and network connection 253.

[0027] In some embodiments, browser 211 is a web browser installed on client 201, and browser 211 is configured with browser monitoring extension 221. Browser monitoring extension 221 is a web browser extension and / or plug-in for monitoring usage of browser 211. In some embodiments, unlike application health monitoring agent 251, browser monitoring extension 221 is configured with permissions that enable collection and monitoring of usage data specific to and / or local to browser 211. For example, usage data associated with accessing SaaS applications via browser 211 may be collected by browser monitoring extension 221. In various embodiments, the usage data collected by browser monitoring extension 221 may include data related to application availability, usage times such as response times, total usage time, DNS lookup times, failed request data, total session time, page views, load times such as page load time, response times, and access times such as last access time, among other usage and operational data. In some embodiments, browser monitoring extension 221 is configured with a list of web applications to monitor, such as a list of websites, domains, and / or application names, and only configured applications have usage data associated with them collected. In various embodiments, the collected usage data is provided to application health monitoring agent 251, which, for example, provides the collected browser-based usage data to an application health monitoring service. In some embodiments, browser 211 uses a network connection (e.g., network connection 213) to access network resources (e.g., web application data). In some embodiments, browser monitoring extension 221 is configured via application health monitoring agent 251, one or more network connections (e.g., network connections 213 and / or 253), and / or another remote configuration interface.

[0028] In some embodiments, application health monitoring agent 251 is an agent application installed on client 201. Application health monitoring agent 251 may be configured to monitor applications and to communicate with browser monitoring extension 221. In various embodiments, application health monitoring agent 251 is further configured to communicate with a cloud-based application health monitoring service via network connection 253. In the illustrated example, application health monitoring agent 251 monitors and collects usage data associated with client 201 and applications running on client 201. For example, application health monitoring agent 251 may collect usage data related to network transfers, such as received and / or transmitted network bytes, CPU usage, memory usage, crashes, error events, restarts, application configuration such as version, installed and / or running applications, application updates, usage based on specific metrics by date, version, user, etc., and / or access data including last access time and total access time, among other usage and operational metrics. In some embodiments, application health monitoring agent 251 collects client metrics regarding application usage, such as CPU usage, memory usage, uptime, system time, disk or storage input / output usage, antivirus functionality, operating system errors and counts, battery health, disk or storage capacity, pending updates, application crashes, and / or metrics related to trigger events such as software, network, and / or hardware events, among other usage and operational metrics.

[0029] FIG. 3 is a block diagram illustrating one embodiment of an application health monitoring service. In the illustrated example, application health monitoring service 301 is a cloud-based service for managing the health of applications used by remote clients. Application health monitoring service 301 communicates with the remote clients via network connection 303 and includes multiple processing modules, including data enrichment module 311, data processing engine 313, aggregation module 315, audit module 317, asset reporting engine 319, and client configuration module 321. In various embodiments, application health monitoring service 301 utilizes one or more data stores (e.g., data store 323). In some embodiments, application health monitoring service 301 is an application health monitoring service used to remotely manage and communicate with application health monitoring service 111 of FIG. 1 and / or clients 101, 103, and 105 of FIG. 1 and / or client 201 of FIG. 2.

[0030] In some embodiments, application health monitoring service 301 includes multiple processing modules for performing one or more different tasks associated with collecting usage data from remote clients. In various embodiments, one or more of the illustrated modules may not be present, and / or additional modules may be present. In some embodiments, the functionality of one or more of the modules may be integrated into a single module or distributed among multiple different modules. In the illustrated example, application health monitoring service 301 receives usage data from remote clients via network connection 303 and provides configuration information to the remote clients using network connection 303. For example, a communication channel is established with the client via network connection 303 to receive collected usage data for the client. Additionally, the communication channel established with the client via network connection 303 may be used to configure application monitoring and data collection for the client, such as which applications' data to collect and the type of data to collect.

[0031] In some embodiments, the data enrichment module 311 is used to enrich received usage data. For example, the data enrichment module 311 may be used to augment, annotate, and / or add further details to usage data collected from clients. In some embodiments, the data enrichment module 311 functions to provide client location data, such as network connection data, such as a network address, and / or geographic data, such as the client's city, subregion, region, state, and / or locale. In some embodiments, the data enrichment module 311 provides further enrichment functionality, such as identifying the client's geographic location through client parameters (e.g., the client's IP address). Other data enrichments may include adding timestamps associated with collected data and calculating latency and throughput metrics for client connections. In various embodiments, data enrichment performed by the data enrichment module 311 may be based on a client identifier (e.g., the client's IP address). Additional metadata not available at the client may be retrieved and / or determined and used to enrich the collected usage data. For example, the client identifier may be used to enrich the client's collected usage data with available licenses assigned to the user of the client device. In some embodiments, the enriched data is based on information aggregated from multiple clients. For example, the usage data may be enriched based on data collected from different clients in the same or different regions (or sub-regions). In various embodiments, once the usage data has been enriched, the enriched data is provided to the data processing engine 313.

[0032] In some embodiments, the data processing engine 313 is used to process the received usage data, including data enriched by the data enrichment module 311. In various embodiments, the data processing engine 313 is a streaming engine and can process a continuous stream of input data in real time. For example, the data processing engine 313 can utilize a high-throughput bus for receiving and processing usage data from multiple clients. In various embodiments, the data is processed and written to one or more data stores (such as data store 323). In some embodiments, the processing includes providing application usage data as raw time series data and / or rolled-up data. For example, rolled-up usage data may be provided by the data processing engine 313 by merging usage data across multiple clients and trimming the merged data to highlight the most relevant metrics. In some embodiments, the processing performed by the data processing engine 313 utilizes additional processing modules (such as the aggregation module 315 and / or the audit module 317).

[0033] In some embodiments, the data processing engine 313 provides an interactive user interface, such as a web dashboard, for viewing processed application usage data. For example, an IT administrator can access the provided dashboard to view application usage data, including raw time series data and rolled-up and / or aggregated usage data. In some embodiments, the interactive dashboard allows the viewer to specify what type of data to view and by what category and / or aggregation parameter (e.g., by user, region, subregion, application, uptime, usage, error rate, resource usage, etc.).

[0034] In some embodiments, the aggregation module 315 is a processing module for aggregating application usage data across multiple clients. For example, the usage data can be analyzed to provide aggregated usage results for a particular application and / or application version (such as the application's average application uptime, the average number of times the application crashes, and the typical usage time of the application). In various embodiments, the data is aggregated over one or more different time windows (such as by minute, day, week, month, or another interval of granularity). In some embodiments, the data is aggregated geographically (such as by region, subregion, network location, etc.). Other forms of aggregation may also be performed, such as by user group, usage metric, department, operating system version, application version, etc.

[0035] In some embodiments, the audit module 317 is a processing module for auditing application usage data. For example, the usage data may be audited with reference to determined or configured usage parameters (such as guaranteed application uptime, quality of service, and / or the number of available purchased licenses). In some embodiments, each application with a guaranteed uptime is analyzed to determine whether the application's actual usage met the uptime expectations. Depending on the results of the audit, one or more actions may be initiated, such as an alert, notification, and / or request for a refund or price change. In some embodiments, usage of the application requires a license that is part of a pool of purchased licenses. The audit module 317 may analyze the application usage data to determine whether the organization purchased the correct or optimal number of licenses. For example, in some scenarios, the organization may purchase more licenses than needed, and the excess licenses may not be utilized. In some embodiments, the excess licenses may be reclaimed and / or returned. As another example, the audit module 317 may determine that the organization did not purchase enough licenses for the application, resulting in some users of the client being unable to access the application because there are no available free licenses. The audit results can identify these deficiencies and initiate actionable solutions to address them, such as by returning unused licenses or initiating a purchase offer for additional licenses. In some embodiments, the audit results are provided to asset reporting engine 319.

[0036] In some embodiments, the asset reporting engine 319 is a processing module for reporting asset audit results. For example, the asset reporting engine 319 may be used to interface with a software asset management (SAM) platform or other software management service for the purpose of reporting discrepancies between actual audit results and predicted results. In some embodiments, the asset reporting engine 319 is used to initiate the return and / or return of unused or unnecessary application licenses. For example, if the audit module 317 determines that a particular application has excessive unused licenses, the asset reporting engine 319 can interface with an appropriate SAM platform to initiate the return of the unused licenses. Similarly, if more licenses are needed, the asset reporting engine 319 can interface with an appropriate SAM platform to purchase additional licenses. In various embodiments, the asset reporting engine 319 can also support corresponding asset-related requests by providing relevant audit information (e.g., analyzed uptime results). In some embodiments, the asset reporting engine 319 is used to generate alerts, notifications, warnings, and / or other reports based on the asset audit results.

[0037] In some embodiments, client configuration module 321 is a processing module for configuring clients for application monitoring. For example, using client configuration module 321, application health monitoring service 301 can provide a user interface for managing remote clients and the health of their applications. In some embodiments, configuration settings are stored by client configuration module 321 in one or more data stores in data stores 323. For example, each client can be configured with an inclusive list of applications to monitor and metrics to collect for the monitored applications.

[0038] In some embodiments, application health monitoring service 301 stores usage data, including processed and analyzed results, in one or more of data stores 323. In some embodiments, data store 323 is a remote database and may include one or more distributed databases and / or one or more time series databases. In various embodiments, application health monitoring service 301 also stores configuration and management settings for clients in data store 323.

[0039] FIG. 4 is a flow chart illustrating one embodiment of a process for monitoring the health of client applications. For example, using the process of FIG. 4, one or more clients can be configured for application health monitoring with an application health monitoring service. In some embodiments, the application health monitoring service is used to configure which applications' usage data to monitor and collect. Once application usage data is collected at the client and provided to the application health monitoring service, the application health monitoring service can analyze the provided usage data, including enriching, aggregating, and performing audits on the data. In various embodiments, the application health monitoring service provides the analyzed usage data as application usage metrics via an interactive user interface dashboard. In some embodiments, the application health monitoring service is application health monitoring service 111 of FIG. 1 and / or application health monitoring service 301 of FIG. 3. In some embodiments, the client device is client 101, 103, and / or 105 of FIG. 1 and / or client 201 of FIG. 2.

[0040] In step 401, a client is configured for application health monitoring. In some embodiments, an IT administrator provides a list of applications to monitor, such as a list of application names, websites, and / or domains, and only the configured applications have usage data associated with them collected. For example, an IT administrator can configure an inclusive list of applications for which usage data should be collected and the type of usage data to collect. Examples of usage data configured for collection may include data related to application availability, response time, usage time such as total usage time, DNS lookup time, failed request data, total session time, page views, load time such as page load time, response time, and access time such as last access time, among other usage and performance data. In some embodiments, usage data collected for installed applications (as compared to browser-based applications) may further include usage data related to network transfers, such as received and / or transmitted network bytes, CPU usage, memory usage, crashes, error events, restarts, application configuration such as version, installed and / or running applications, application updates, usage based on specific metrics by date, version, user, etc., and / or access data including last access time and total access time, among other usage and operational metrics.

[0041] In various embodiments, the configuration parameters are provided to the client via an agent and / or browser extension installed on the client that performs application monitoring and data collection. In some embodiments, the agent is application health monitoring agent 251 of FIG. 2, and the browser extension is browser monitoring extension 221 of FIG. 2. For example, once the configuration is provided by an administrator to the application health monitoring service, the application health monitoring service can provide the client configuration to the appropriate client over a network channel established with the application health monitoring agent installed on the client device. In some embodiments, the application health monitoring agent then communicates with the browser monitoring extension to configure the browser extension for monitoring browser-based applications.

[0042] At step 403, client usage data is received. For example, application usage data collected at a client is received. In some embodiments, the data is streamed to an application health monitoring service via an application health monitoring agent installed on the client. The received streaming data may correspond to application data from multiple clients and may be processed via a high-throughput performance bus for receiving and processing usage data from multiple clients.

[0043] At step 405, the usage data is analyzed. For example, the usage data may be analyzed by enriching the data with additional information based on the client and / or collection process. In some embodiments, the usage data is analyzed by aggregating data collected across multiple clients, such as aggregating data based on categories such as different time periods, user groups, locations or regions, availability, etc. In various embodiments, the usage data is further analyzed by auditing the data to identify whether usage meets expected constraints (such as the number of available licenses and / or application uptime requirements). The analyzed data may be provided as aggregated data, as raw time series data, as rolled-up data, and / or using another form of analyzed data results. In various embodiments, the usage data is analyzed to identify service degradation, service outages, and / or other specific patterns (such as regional patterns based on data aggregated at a subregion level).

[0044] At step 407, the usage data results are provided. For example, the usage data results may be provided as an interactive user interface dashboard to review and identify application usage trends, such as, for example, performance and availability trends. As another example, the usage data may be provided as notifications and / or audit reports to identify application performance deficiencies, including under- and over-subscription of licenses, degradation of service quality, and / or service outages. For example, the usage data may be used to redeem and / or return unused licenses or to purchase needed additional licenses. In some embodiments, the usage data is provided to a software asset management platform to ascertain usage and performance characteristics (e.g., license usage and application uptime) of the monitored application.

[0045] FIG. 5 is a flow chart illustrating one embodiment of a process for configuring a client for application health monitoring. For example, using the process of FIG. 5, one or more clients may be configured for application health monitoring with an application health monitoring service. In some embodiments, the application health monitoring service is used to configure the client, including initiating installation of an application health monitoring agent and / or browser monitoring extension on the client to collect application usage data. In some embodiments, the process of FIG. 5 is performed in step 401 of FIG. 4. In some embodiments, the application health monitoring service is application health monitoring service 111 of FIG. 1 and / or application health monitoring service 301 of FIG. 3. In some embodiments, the client device is client 101, 103, and / or 105 of FIG. 1 and / or client 201 of FIG. 2.

[0046] In step 501, an application health monitoring agent is installed on a client. For example, a software application health monitoring agent is installed on the client with access rights that enable the agent to both monitor applications running on the client and retrieve client device information. The agent is further configured with access privileges to establish and / or accommodate a network connection between the agent on the client and a cloud-based application health monitoring service. For example, the installed agent can collect application usage metrics and provide them to the cloud-based application health monitoring service over the network connection. In some embodiments, the installed agent communicates with one or more browser monitoring extensions to relay additional data collected by the browser extensions to the cloud-based application health monitoring service. The installed agent can also be used to update application monitoring configuration parameters, such as the configuration of the browser monitoring extensions. In some embodiments, the application health monitoring agent is application health monitoring agent 251 of FIG. 2.

[0047] At step 503, a browser monitoring extension is installed on the client. For example, a browser monitoring extension (or browser plug-in) is installed on the client and integrated with the client's browser software. The browser monitoring extension is configured to enable the browser monitoring extension to monitor browser-based applications accessed via the client's web browser. For example, the browser monitoring extension can monitor websites and / or web applications accessed by the client via the browser and collect usage metrics (such as response time, session time, page views, and page load time, among other metrics). In some embodiments, one or more different browser monitoring extensions are installed for each different browser configured for the client. For example, different browser monitoring extensions or different versions may be installed and / or customized for different web browsers. In some embodiments, the installed browser monitoring extension communicates with the client agent installed in step 501, for example, to receive configuration updates and / or relay collected application usage data. In some embodiments, the browser monitoring extension is browser monitoring extension 221 of FIG. 2.

[0048] At step 505, applications are configured for health monitoring. For example, a list of applications for which usage data is to be monitored and collected is provided, for example, by an IT administrator. The list of applications may specify whether the applications are installed applications running natively on the client or web-based or software-as-a-service (SaaS) applications accessed via a client browser. In various embodiments, the configured applications may be identified by application name, website, domain name, domain wildcard, or another identification format. In some embodiments, the applications to be monitored are based on an inclusion list, and applications must be added to the inclusion list in order for their usage to be monitored. In some embodiments, a separate or default configuration for application monitoring is used. For example, default application monitoring settings may be configured during the client provisioning process. In various embodiments, the configuration for application monitoring may include specifying which usage data and metrics are to be collected and how they are to be collected (e.g., parameters associated with the collection, such as frequency, resolution, format, etc.). For example, an IT administrator can specify the application metrics and data to be collected and how often the data is to be collected. In some embodiments, the configuration parameters are provided and / or managed by an administrator via a cloud-based application health monitoring service and then relayed to the installed agent and / or browser extension. For example, the agent installed in step 501 can receive the configuration parameters, update the agent's configuration, and further provide the browser-based configuration parameters to the installed browser monitoring extension so that the browser monitoring extension can update its configuration parameters.

[0049] At step 507, the configured applications are monitored. For example, the applications configured for monitoring at step 505 are monitored for their usage. In various embodiments, the installed application health monitoring agent and browser monitoring extension each identify when an application configured for monitoring is active and begin collecting usage data for the running application. For example, the application health monitoring agent can monitor processes running on the client, and if a process matches an application configured for application health monitoring, usage metrics associated with the matched application are collected. Similarly, the browser monitoring extension can monitor active websites accessed through the browser, and if a website matches a web application configured for application health monitoring, usage metrics associated with the matched web application are collected. In some embodiments, once metrics are collected, they are stored locally until they can be provided to a cloud-based application health monitoring service. In some embodiments, the agent can also collect usage metrics and data and receive metrics collected by the browser monitoring extension.

[0050] At step 509, a communication channel with the monitoring service is activated. For example, a network connection is established between the client and the cloud-based application health monitoring service. In some embodiments, the network connection is established with an installed application health monitoring agent running on the client. For example, a communication channel is established with the installed agent to provide the collected usage data stored in the agent to the cloud-based application health monitoring service. In some embodiments, a communication channel with the monitoring service is also activated between the cloud-based application health monitoring service and a browser monitoring extension. For example, in some embodiments, the browser monitoring extension can provide the collected browser application usage data directly to the cloud-based service instead of through an installed agent.

[0051] FIG. 6 is a flowchart illustrating one embodiment of a process for analyzing collected application health data. For example, using the process of FIG. 6, a cloud-based application health monitoring service receives and analyzes application usage data provided by clients to determine application health results. In various embodiments, the received usage data is enriched as part of the analysis process and then further analyzed to provide aggregated and / or audited data results. The analyzed application health results may be provided via an interactive user interface dashboard and may initiate software asset management actions (e.g., returning unused licenses). In some embodiments, the process of FIG. 6 is performed by the application health monitoring service at steps 403 and / or 405 of FIG. 4. In some embodiments, the application health monitoring service is application health monitoring service 111 of FIG. 1 and / or application health monitoring service 301 of FIG. 3. In some embodiments, the client devices providing the usage data are clients 101, 103, and 105 of FIG. 1 and / or client 201 of FIG. 2.

[0052] In step 601, client usage data is received. For example, usage data collected by a client is provided to the application health monitoring service via an activated communication channel. The received usage data may include usage data associated with the application being monitored and client device data. In various embodiments, the usage data is received as a stream of data and processed via a high-throughput performance bus. In some embodiments, data is received from multiple clients, such as by intermingling different client data.

[0053] At step 603, the usage data is enriched. For example, the usage data is enriched with additional relevant data provided by sources other than the client. In some embodiments, the data is enriched based on a client and / or application identifier (e.g., the client's IP address). For example, a location may be determined for the client based on the client's network address, and the usage data is enriched with location and / or geographic data, such as the client's city, region, state, and / or locale. Other data enrichments may include adding timestamps associated with the collected data and calculating latency and throughput metrics for the client connection. Additional metadata not available at the client may be retrieved and / or determined and used to augment, annotate, and / or add further detail to the collected usage data. For example, a client identifier may be used to enrich the client's collected usage data with available licenses assigned to the user of the client device. In some embodiments, the enriched data is based on information aggregated from multiple clients. For example, the usage data may be enriched based on data collected from different clients in the same or different regions.

[0054] At step 605, the usage data is aggregated. For example, raw data may be aggregated across multiple usage data updates using data received from multiple clients. For example, for a particular application and / or application version, the usage data may be analyzed to provide aggregated usage results (such as the application's average application uptime, the average number of crashes for the application, and the typical usage duration of the application). In various embodiments, the data is aggregated over one or more different time windows (such as daily, weekly, monthly, etc.). In some embodiments, the data is aggregated geographically (such as by region, sub-region, network location, etc.). For example, data may be aggregated by sub-region to identify geographic patterns. Other forms of aggregation may also be performed, such as by user group, usage metric, department, operating system version, application version, etc. In some embodiments, the aggregated data results include processing the raw data as time series data to generate rolled-up data results. For example, rolled-up usage data may be provided by merging usage data across multiple clients and trimming the merged data to highlight the most relevant metrics.

[0055] At step 607, an audit is performed using the usage data. For example, the usage data is audited with reference to one or more determined or configured usage parameters (such as guaranteed application uptime, guaranteed quality of service, and / or number of available purchased licenses, among others). For example, the data may be audited to identify usage patterns, such as analyzing aggregated data at a sub-region level to identify specific patterns at a regional level. In some embodiments, each application with a guaranteed uptime is analyzed to determine whether the application's actual usage met the uptime expectations. Other forms of audit analysis may also be performed. For example, for a particular application, use of the application by a client requires the client to first obtain an application license that is part of a pool of purchased licenses. An audit analysis of application usage determines whether an organization has purchased the correct or optimal number of licenses. In some situations, an organization ends up purchasing more licenses than needed, and the excess licenses go unused. The audit analysis identifies these discrepancies. As another example, the audit analysis may determine that an organization has not purchased enough licenses for a particular application, resulting in some users of clients being unable to access the application because there are no free licenses available. Audit results can identify these deficiencies.

[0056] At step 609, the analysis results are stored. For example, various forms of usage data and analysis results (e.g., application health results) are stored in one or more data stores. In some embodiments, the usage data and results are stored and made available for review via an interactive user interface dashboard. For example, raw time series data, rolled-up data, aggregated data, and audited data, and corresponding results, may be stored and made available for review via a dashboard. In some embodiments, once an aggregation parameter is reached, the usage data and results are stored for further aggregation. For example, data may be aggregated over a month, and the data is stored until at least one month's worth of data has been collected.

[0057] FIG. 7 is a flow chart illustrating one embodiment of a process for providing application health results. For example, using the process of FIG. 7, a cloud-based application health monitoring service provides results from an analysis of application usage data in the form of an interactive user interface dashboard. The provided results may include aggregated and / or summarized application data metrics for applications being used by clients under the management of the application health monitoring service. In some embodiments, the process of FIG. 7 is performed by the application health monitoring service at step 407 of FIG. 4. In some embodiments, the provided usage metrics are processed using the analysis described with respect to the process of FIG. 6. In some embodiments, the application health monitoring service is application health monitoring service 111 of FIG. 1 and / or application health monitoring service 301 of FIG. 3. In some embodiments, the client devices providing the usage data are clients 101, 103, and 105 of FIG. 1 and / or client 201 of FIG. 2.

[0058] At step 701, a request for application usage metrics is received. For example, a request for application metrics collected and analyzed from a client under the management of the application health monitoring service is provided by a client (e.g., a network client) to the application health monitoring service. In some embodiments, the request is received via a dashboard provided by the application health monitoring service for interacting with application health metrics. The request may specify usage metrics to display by configuring the request for a particular client, application, and / or relevant parameters of the client and / or application (e.g., location, geography, user group, uptime, utilization rate, error rate, resource utilization, update status, license underutilization, license overutilization, etc.).

[0059] At step 703, analyzed usage metrics are retrieved. For example, metrics matching the usage metrics requested at step 701 are retrieved from one or more cloud-based data stores. In some embodiments, the analyzed metrics are enriched data metrics that include aggregated and / or rolled-up data results. For example, the retrieved metrics may include application usage metrics aggregated for a particular time frame. The analyzed metrics may further include audit results, such as results from an audit of application usage for uptime assurance and / or license usage.

[0060] At step 705, the requested metrics are provided via a dashboard. For example, based on the request received at step 701, the metrics retrieved at step 703 are provided via an interactive user interface dashboard. In some embodiments, the dashboard is a web-based graphical user interface that allows the viewer to submit subsequent requests to zoom out or drill down on the data and / or modify the request. For example, drilling down on the data may allow the viewer to request more detailed metrics associated with a finer set of clients, users, locations, and / or applications. In some embodiments, the dashboard also allows the viewer to provide different time frames for the aggregated results and to change how the results are aggregated, such as for different elements such as locations, clients, and applications. For example, results aggregated by month may be drilled down to provide results aggregated by week and by day. In some embodiments, the information provided in the audit report includes application usage metrics with reference to uptime guarantees and / or license usage with reference to purchased licenses. The provided information may also include alerts, warnings, notifications, and / or other messages to alert the user to identified usage anomalies (e.g., a particular application being out of service, etc.). In some embodiments, one or more of the provided usage metrics may be determined in real time based on the usage metrics retrieved in step 703.

[0061] FIG. 8 is a flow chart illustrating one embodiment of a process for identifying and returning unnecessary application licenses. For example, the process of FIG. 8 can be used to identify insufficient application license utilization and initiate the return of unnecessary application licenses. By auditing application utilization data for license utilization, a cloud-based application health monitoring service can initiate the license return process in conjunction with a software asset management (SAM) service corresponding to the application. While this process is described with respect to licenses, a similar process may be performed with respect to other application metrics, such as auditing application uptime with respect to an application uptime guarantee or service contract. In some embodiments, the process of FIG. 8 is performed by the application health monitoring service at step 407 of FIG. 4. In some embodiments, the application health monitoring service is application health monitoring service 111 of FIG. 1 and / or application health monitoring service 301 of FIG. 3. In some embodiments, the client device providing the utilization data is clients 101, 103, and 105 of FIG. 1 and / or client 201 of FIG. 2.

[0062] At step 801, an application operating outside of its license usage parameters is identified. For example, an application may be configured with license usage parameters that trigger when the application is underutilizing its licenses (i.e., having more licenses available than needed). In some embodiments, an underutilization event may be triggered using various evaluation techniques (e.g., triggers) based on the number or percentage of unused but available licenses, the time a certain percentage of licenses are utilized, the time a certain number of licenses are available, etc. In various embodiments, various underutilization scenarios may be configured and utilized to trigger when an application is operating outside of its license usage parameters. At step 801, application usage data is audited to identify applications that do not meet the configured license usage parameters and the number of licenses that should be returned. In some embodiments, a time period associated with the license return is also identified. For example, the identified time period may be for a past period and / or for future usage of the application.

[0063] In step 803, a software management service for the application is contacted. For example, a software management service (such as a software asset management (SAM) service) responsible for managing licenses for the application is identified and contacted. In some embodiments, the connection is established using a well-defined application programming interface (API) or via another established communication protocol.

[0064] At step 805, a reclamation request is provided. For example, a reclamation request is requested for unused licenses. In some embodiments, the request is for licenses that have not been used in the past based on previous usage history. For example, the reclamation request may be for licenses purchased that have not been used in the past month. In some embodiments, the request is based on future projected application usage, where future demand for licenses is based on past usage of the application. In various embodiments, the reclamation request identifies the application and the number of licenses being requested to be reclamated.

[0065] At step 807, audit support for the reclaim is provided. For example, the software management service contacted at step 803 may request the reclaim support provided by step 805. At step 807, the results of the audit analysis performed on the collected client usage data are provided as audit support for the reclaim. The audit support provided may include time-stamped application usage data collected from application health monitoring. In some embodiments, the audit support includes usage data enriched with additional client descriptions (such as client user, location, and geographic information).

[0066] FIG. 9 is a functional diagram illustrating a computer system programmed to monitor application health. Clearly, other computer system architectures and configurations may be utilized to maintain the obfuscation order of the protected data sets and / or perform comparison queries on the obfuscated data. Examples of computer system 900 include clients 101, 103, and 105 of FIG. 1, client 201 of FIG. 2, one or more computers of application health monitoring service 111 of FIG. 1, and / or one or more computers of application health monitoring service 301 of FIG. 3. Computer system 900, which includes various subsystems as described below, includes at least one microprocessor subsystem (also referred to as a processor or central processing unit (CPU)) 902. For example, processor 902 may be implemented by a single-chip processor or a multiprocessor. In some embodiments, processor 902 is a general-purpose digital processor that controls the operation of computer system 900. Using instructions retrieved from memory 910, processor 902 controls the receipt and manipulation of input data and the output and display of data on an output device (e.g., display 918). In various embodiments, one or more computer systems 900 may be utilized to implement at least portions of the processes of Figures 4-8 and the interactive user interface dashboards of Figures 10 and / or 11.

[0067] The processor 902 is bidirectionally coupled to memory 910, which may include a first primary storage area (typically random access memory (RAM)) and a second primary storage area (typically read-only memory (ROM)). As known to those skilled in the art, the primary storage may be used as a general storage area, a scratchpad memory, and for storing input data and processed data. The primary storage may also store programming instructions and data in the form of data objects and text objects, in addition to other data and instructions for processes executed on the processor 902. As also known to those skilled in the art, the primary storage typically comprises basic operating instructions, program code, data, and objects used by the processor 902 to execute functions (e.g., programmed instructions). For example, the memory 910 may include any suitable computer-readable storage medium, as described below, depending, for example, on whether data access needs to be bidirectional or unidirectional. For example, the processor 902 may directly and very quickly store and retrieve frequently needed data from a cache memory (not shown).

[0068] A removable mass storage device 912 provides additional data storage capacity for the computer system 900 and is connected bidirectionally (read / write) or unidirectionally (read only) to the processor 902. For example, storage 912 may include computer-readable media such as magnetic tape, flash memory, PC cards, portable mass storage devices, holographic storage devices, and other storage devices. Fixed mass storage 920 may also provide additional data storage capacity, for example. The most common example of mass storage 920 is a hard disk drive. Mass storage 912, 920 generally stores additional programming instructions, data, etc. that are not typically utilized by the processor 902. It will be understood that the information maintained in mass storage 912 and 920 may, if desired, be incorporated in standard fashion into part of memory 910 (e.g., RAM) as virtual memory.

[0069] In addition to providing processor 902 with access to storage subsystems, bus 914 may also be used to provide access to other subsystems and devices. As shown, these may include a display monitor 918, a network interface 916, a keyboard 904, and a pointing device 906, as well as auxiliary input / output device interfaces, a sound card, speakers, and other subsystems, as needed. For example, pointing device 906 may be a mouse, stylus, trackball, or tablet, and is useful for interacting with a graphical user interface.

[0070] The network interface 916 allows the processor 902 to be connected to another computer, computer network, or telecommunications network using a network connection, as shown. For example, through the network interface 916, the processor 902 can receive information (e.g., data objects or program instructions) from another network or output information to another network in the course of performing a method / processing step. Information, often represented as a series of instructions executed on the processor, can be received from and output to another network. An interface card (or similar device) and appropriate software implemented (e.g., executed / performed) by the processor 902 can be used to connect the computer system 900 to an external network and transfer data according to standard protocols. For example, various processing embodiments disclosed herein may be executed on the processor 902 or may be executed over a network (such as the Internet, an intranetwork, or a local area network) with a remote processor that shares some of the processing. Additional mass storage devices (not shown) may be connected to the processor 902 through the network interface 916.

[0071] Auxiliary I / O device interfaces (not shown) may be used with computer system 900. Auxiliary I / O device interfaces may include generic and customized interfaces that allow processor 902 to send data to, and more typically receive data from, other devices such as microphones, touch-sensitive displays, transducer card readers, tape readers, voice or handwriting recognition devices, biometric readers, cameras, portable mass storage devices, and other computers.

[0072] Additionally, various embodiments disclosed herein further relate to computer storage products including a computer-readable medium having program code thereon for performing various computer-implemented operations. A computer-readable medium is any data storage device that can store data, which can thereafter be read by a computer system. Examples of computer-readable media include, but are not limited to, all of the above media: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media such as optical disks; and specially configured hardware devices such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and ROM / RAM devices. Examples of program code include, for example, machine code produced by a compiler, or files containing high-level code (e.g., scripts) that can be executed using an interpreter.

[0073] The computer system illustrated in Figure 9 is but one example of a computer system suitable for use with various embodiments disclosed herein. Other computer systems suitable for such use may include more or fewer subsystems. Furthermore, bus 914 is an example of any interconnection scheme that functions to link the subsystems. Other computer architectures having different configurations of subsystems may also be used.

[0074] FIG. 10 illustrates one embodiment of a user interface for viewing analyzed usage metrics for multiple applications via an interactive user interface dashboard. In the illustrated example, user interface 1000 is a user interface view of an application health dashboard provided by an application health monitoring service. User interface 1000 includes and displays analyzed application usage metrics, such as aggregated metrics collected from monitored clients. As shown in FIG. 10, the aggregated application metrics include active users, active devices, impacted users, and monitored applications. The top issues identified from the collected usage data are shown along with the corresponding applications and aggregated details of those corresponding issues. The bottom of user interface 1000 provides a geographical representation of applications and their corresponding aggregated user experiences with colors coded to different user client experiences. In some embodiments, user interface 1000 is provided by an application health monitoring service (e.g., application health monitoring service 111 of FIG. 1 and / or application health monitoring service 301 of FIG. 3). In some embodiments, the usage data includes data collected from a client (such as clients 101, 103, and / or 105 of FIG. 1 and / or client 201 of FIG. 2) using the processes of FIGS. 4-8.

[0075] FIG. 11 illustrates one embodiment of a user interface for viewing analyzed usage metrics collected from multiple users for a particular application. In the illustrated example, user interface 1100 is a user interface view of an application health dashboard provided by an application health monitoring service. User interface 1100 includes and displays analyzed application usage metrics, such as aggregated metrics collected from monitored clients for the Software as a Service (SaaS) application Office 365. As shown in FIG. 11 , the aggregated application metrics include Total users, Sessions, Page Views, Average Session Duration, Average response time, Average page load time, Failed web requests, Total incidents, Total alerts, and Impacted uses. The bottom of user interface 1100 displays a user interface dialog that allows the viewer to view network information for different employees who used the monitored application. The network information includes enriched usage data, such as associated network addresses and corresponding network hop times. In some embodiments, user interface 1100 is provided by an application health monitoring service (such as application health monitoring service 111 of FIG. 1 and / or application health monitoring service 301 of FIG. 3). In some embodiments, the usage data includes data collected from a client (such as clients 101, 103, and / or 105 of FIG. 1 and / or client 201 of FIG. 2) using the processes of FIGS. 4-8.

[0076] Although the above embodiments have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and are not intended to be limiting.

Claims

1. 1. A method comprising: The processor activates a communication channel with an agent installed on the client; while the communication channel is active, the processor collects usage data via the agent, the usage data characterizing usage for a plurality of different applications accessed via the client; the processor analyzing the usage data to determine metrics associated with the plurality of different applications; the processor providing an interactive user interface dashboard presenting the determined metrics; the processor providing a request for the return of one or more licenses to one of the plurality of different applications; A method comprising:

2. 10. The method of claim 1, further comprising the processor determining one or more physical locations associated with the client.

3. 10. The method of claim 1, wherein the collected usage data includes at least one of response time, session time, page load time, or last access time.

4. 10. The method of claim 1, wherein the determined metrics associated with the plurality of different applications include at least one of an overall user metric, a session metric, a page view metric, an average session duration metric, an average response time metric, an average page load time metric, or a request failure metric.

5. The method of claim 1 , wherein the collected usage data includes at least one of CPU utilization, memory utilization, application version, or last access time.

6. 2. The method of claim 1, wherein the determined metrics associated with the plurality of different applications include at least one of active user metrics, active device metrics, or monitored application metrics.

7. The method of claim 1 , wherein the determined metrics associated with the plurality of different applications are grouped by geographic region.

8. The method of claim 1 , wherein the plurality of different applications comprises a combination of one or more user device applications and one or more cloud services.

9. The method of claim 1 , wherein each of the plurality of different applications corresponds to a third-party application.

10. 1. A system comprising: one or more processors; a memory coupled to the one or more processors; Equipped with The memory, when executed, causes the one or more processors to: Activates a communication channel with the agent installed on the client, collecting usage data via said agent while said communication channel is active; the usage data characterizing usage for a plurality of different applications accessed via the client; analyzing the usage data to determine metrics associated with the plurality of different applications; the determined metrics include audit data and uptime data regarding license utilization; A system configured to provide instructions to the one or more processors to cause the one or more processors to provide an interactive user interface dashboard that presents the determined metrics.

11. 11. The system of claim 10, further comprising determining one or more physical locations associated with the client.

12. 11. The system of claim 10, wherein the collected usage data includes at least one of response time, session time, page load time, or last access time.

13. 11. The system of claim 10, wherein the determined metrics associated with the plurality of different applications include at least one of an overall user metric, a session metric, a page view metric, an average session duration metric, an average response time metric, an average page load time metric, or a request failure metric.

14. 11. The system of claim 10, wherein the collected usage data includes at least one of CPU utilization, memory utilization, application version, or last access time.

15. 11. The system of claim 10, wherein the determined metrics associated with the plurality of different applications include at least one of active user metrics, active device metrics, or monitored application metrics.

16. The system of claim 10 , wherein the determined metrics associated with the plurality of different applications are grouped by geographic region.

17. 11. The system of claim 10, wherein the plurality of different applications comprises a combination of one or more user device applications and one or more cloud services.

18. 11. The system of claim 10, wherein each of the plurality of different applications corresponds to a third-party application.

19. A computer program product embodied in a non-transitory computer-readable storage medium, computer instructions for activating a communication channel with an agent installed on the client; computer instructions for collecting usage data via the agent while the communication channel is active, the usage data characterizing usage for a plurality of different applications accessed via the client; and computer instructions for analyzing the usage data to determine metrics associated with the plurality of different applications; computer instructions for providing an interactive user interface dashboard that presents the determined metrics; computer instructions for providing a request to return one or more licenses to one of the plurality of different applications; A computer program product comprising:

20. The method of claim 1, further comprising: the processor identifying one or more applications from the plurality of different applications that include a guaranteed application uptime; the processor analyzing the usage data to determine usage of the one or more applications; initiating a refund request in response to the processor determining that usage of the one or more applications does not meet the guaranteed application uptime; A method comprising:

Images