Real-time event data reporting on edge computing devices

Abstract

The present disclosure relates to utilizing a real-time event data reporting system that enables real-time and near-real-time monitoring and reporting in edge devices. For example, in various examples, the real-time event data reporting system embeds a service within a conventional event data collector of an edge device to obtain, organize, and publish event data for a local computing device in real-time using memory storage. Additionally, the real-time event data reporting system also processes the published event data to generate aggregated data that can be persisted to persistent storage. In this manner, the real-time reporting system efficiently and accurately provides event data reports and processed indicator data to a client device in real time or near real time while utilizing additional fallback safeguards. Indeed, the real-time reporting system provides a highly available, fault tolerant, distributed, scalable, and efficient mechanism for collecting and managing various metrics from services in an edge or cloud environment.

Description

Background Art

[0001] In recent years, computing devices have made significant progress in both hardware and software. For example, more and more entities are beginning to use the services provided by cloud computing systems. In order to improve some of the problems associated with using cloud computing system services, some existing computer systems have adopted edge computing devices (e.g., Stack Edge devices), which can provide services with lower latency, higher bandwidth and / or stronger connectivity. For example, edge computing devices (or "edge devices") can provide hardware as a service (HWaaS) solutions, bringing the power of cloud computing systems to an entity's local network (e.g., an enterprise internal (on-prem) network). In fact, edge devices can provide localized processing for an entity for services requested by other computing devices running within the entity's local network. Summary of the invention

[0002] The systems and methods of the present disclosure solve many problems in the art by enabling real-time and near real-time monitoring and reporting in edge devices. For example, the systems and methods of the present disclosure provide an edge device framework that can collect and manage event data indicators from local network services in real-time or near real-time in an efficient, highly available, fault-tolerant, distributed, and scalable manner.

[0003] For example, in one or more implementations, the system and method of the present disclosure embeds a service into a traditional event data collector of an edge device, and uses memory storage to acquire, organize, and publish event data of a local computing device in real time. In addition, the system and method of the present disclosure further processes the published event data, generates aggregated data, and persists it to persistent storage, which enables the processed event data to be quickly provided to client devices, such as administrator devices.

[0004] As described in detail below, the systems and methods of the present disclosure have many advantages over existing computer systems, such as facilitating real-time or near real-time monitoring of local networks and systems, and reducing processing latency of indicators. Additionally, the systems and methods of the present disclosure utilize hierarchical aggregation, which reduces the number of requests and enables edge devices to report when disconnected from the corresponding cloud computing system. Further, the systems and methods of the present disclosure can be integrated with any service at any scale and are platform-independent. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The detailed description provides additional specificity and details for one or more implementations through the use of the accompanying drawings, which are briefly described as follows:

[0006] Figure 1A-1BA schematic diagram of an example computing system environment is shown in which a real-time event data reporting system is implemented according to one or more implementations.

[0007] Figure 2 An example overview of implementing a real-time event data reporting system and a fallback mechanism according to one or more implementations is shown.

[0008] Figure 3 A schematic diagram illustrating example actions and operations performed by a real-time event data reporting system to improve event data reporting according to one or more implementations.

[0009] Figure 4 A schematic diagram of an example method for publishing an event graph from received event data using an event data collector within a real-time event data reporting system according to one or more implementations is shown.

[0010] Figure 5 A diagram illustrating an example method for persisting collated data to persistent storage using a monitoring agent within a real-time event data reporting system according to one or more implementations.

[0011] Figure 6 A sequence diagram is shown of an example method for establishing and utilizing a backoff flag to protect against edge device failures in accordance with one or more implementations.

[0012] Figure 7 A schematic diagram of an example method for preventing edge device loss by using a virtual edge device group according to one or more implementations is shown.

[0013] Figure 8 An example graphical user interface that displays an event data report based on persisted event data is shown in accordance with one or more implementations.

[0014] Fig. 9 An example series of actions is shown for persisting monitored and processed event data to persistent storage in accordance with one or more implementations.

[0015] Fig.10 Example components included in a computer system are shown. DETAILED DESCRIPTION

[0016] Implementations of the present disclosure provide benefits and / or solve one or more of the above-mentioned or other problems in the art through systems, computer-readable media, and methods, which utilize a real-time event data reporting system that enables real-time and near real-time monitoring and reporting in edge devices. For example, in various implementations, a real-time event data reporting system (or "real-time reporting system") embeds a service into a traditional event data collector of an edge device to acquire, organize, and publish event data for a local computing device in real time using memory storage.

[0017] According to the information provided in this document, a real-time reporting system provides a real-time monitoring framework that facilitates a fault-tolerant, efficient method to collect, process, and store metrics on edge devices. In particular, as described in detail below, the real-time reporting system facilitates efficient collection and processing of metrics, automatic configuration of multi-level aggregation of metrics, custom metric validation, custom retention processing, and virtual grouping of edge devices. Additionally, the real-time reporting system runs on devices ranging from memory-constrained edge devices to large cloud computing systems.

[0018] To elaborate, in one or more implementations, the real-time reporting system further processes the published event data to generate aggregated data and persists it to persistent storage. In this way, the real-time reporting system is able to efficiently and accurately provide event data reports with processed metric data to client devices, either in real-time or near real-time when additional fallback protection measures are used. Indeed, the real-time reporting system provides a highly available, fault-tolerant, distributed, scalable, and efficient mechanism for collecting and managing various metrics from services in edge or cloud environments.

[0019] For illustration, in one or more implementations, the real-time reporting system detects event data generated by a data source in real time, and these event data will be written to an event record (e.g., a service log, an audit log, or a database log). In response, the real-time reporting system stores a copy of the event data in a non-persistent memory in real time in an event group containing an event map (eventmap). Additionally, the real-time reporting system also publishes the event map with a unique event group identifier to a distributed queue in real time as published event data. Further, the real-time reporting system processes the published event data based on the unique event group identifier by reading the published event data from the distributed queue into the consumer event map, and persists the processed event data from the consumer event map to the persistent storage, thereby generating processed event data. The real-time reporting system also accesses the processed event data from the persistent storage and provides it to a computing device (e.g., for display) as report data.

[0020] As described herein, the real-time reporting system provides several technical advantages in terms of computational efficiency, accuracy, and flexibility compared to existing computing systems. Indeed, the real-time reporting system provides several practical applications that provide benefits and / or solve problems associated with real-time edge device metric monitoring and reporting.

[0021] To illustrate the technical advantages of real-time reporting systems, consider the significant inefficiency, inaccuracy, and inflexibility of existing computer systems in monitoring and reporting edge device metrics in local networks. In detail, existing computer systems using edge devices are typically inefficient in monitoring, collecting, and reporting important information and metrics (e.g., event data, system data, and other data) reported by local computing devices within a local network. For example, most existing computer systems report metrics asynchronously, rather than in real time. Instead, these existing computer systems are agent-based and rely on system processes to slowly collect metrics and then publish the metrics by sending batches to cloud computing systems at delayed intervals.

[0022] Additionally, in many instances, the cloud system remotely aggregates the received data, and upon receiving a query from a client device in the same local network as the edge device, the cloud system returns the processed event data to the local network. However, either way, sending data from the local network to the cloud system and returning the processed event data to the local network may require a large number of hops, which introduces latency and subjects the data to several network errors.

[0023] Further, many existing computer systems do not aggregate data in a useful format. Instead, these existing computer systems often provide unorganized data that requires computing resources and time before useful information can be extracted. Similarly, existing computer systems often suffer from wasted computing resources using too many event data reports and requests.

[0024] In addition, existing computer systems are often inaccurate. For example, in many implementations, existing computer systems inefficiently report and process duplicate data, which not only wastes bandwidth and computing resources (e.g., processing power and memory), but can also lead to inaccuracies when aggregating data. For example, duplicate data entries can distort the total memory usage results of a local device.

[0025] To illustrate another example of inaccuracy, local networks are often disconnected from the corresponding cloud computing system and / or the public Internet. When a local network is disconnected from a cloud computing system, existing computer systems are unable to monitor and report metrics. As a result, administrators associated with the local network are often unable to accurately discover ongoing activities in the local network or the status of local computing devices.

[0026] Furthermore, existing computer systems are not flexible. For example, many existing computer systems are fixed to very specific operating systems and expect data from a certain number of devices. Additionally, some existing computer systems require a large amount of computing resources (e.g., processing power, memory, and storage). These two limitations typically limit the number of devices that can be monitored in a local network.

[0027] As described above, in various implementations, the real-time reporting system improves efficiency by providing real-time reporting of indicators by edge devices for local networks (e.g., an internal enterprise network operated by an entity). For example, in various implementations, unlike agent-based systems, the real-time reporting system captures and stores real-time event data (e.g., service event data intended to be written to a system-based service log or database log) in a memory (e.g., non-persistent or volatile memory) queue for quick and efficient publishing. In this way, the real-time reporting system enables the published data to be quickly processed and stored in persistent storage for quick access by user client devices (e.g., client devices associated with a user). Unlike existing computer systems, the real-time reporting system does not suffer large delays when sending event data from an edge device to a cloud computing system, waiting for the cloud computing system to process the data, and / or sending the data back to the edge device.

[0028] As another example, the real-time reporting system efficiently stores event data into groups such as event graphs and event graph groups. Therefore, the real-time reporting system reduces the number of data storage and requests, which not only improves the publishing and processing process, but also reduces the use of bandwidth. In various implementations, the real-time reporting system utilizes event graphs and event graph groups to improve communication efficiency without introducing latency into the system. Therefore, the number of storage requests and reports to the edge device system is reduced. In fact, the real-time reporting system utilizes a hierarchical aggregation model to aggregate data at different levels, which reduces the number of calls and requests between systems and components. For example, in addition to the processing that may be performed at each level, in the hierarchical aggregation model, the received indicators are aggregated into event graphs, which are aggregated into published event groups, which are then aggregated into processed data. In addition, having aggregated data enables the real-time reporting system to continue to operate when disconnected from the cloud service.

[0029] Additionally, by efficiently publishing, processing, and storing event data metrics, the real-time reporting system reduces duplicate publishing and duplicate processing. For example, even if data is reported multiple times, the real-time reporting system only publishes the data once. In various implementations, the real-time reporting system utilizes unique keys in a key-value store to help process and filter out previously processed data. Likewise, in many implementations, the real-time reporting system processes data only once to reduce wasteful processing of redundant data.

[0030] As described above, the real-time reporting system improves the accuracy of reporting and processing of event data beyond existing computer systems. For example, in many implementations, the real-time reporting system provides idempotent processing, eliminating inaccurate aggregation and processing of event data. In addition, in various implementations, the real-time reporting system utilizes event graphs aggregated in a time series format and / or unique keys to help avoid duplicate processing. For example, the real-time reporting system utilizes these tools to filter out previously processed data, which may lead to inaccurate operations and results.

[0031] As another example, by performing improved fault-tolerant operations, the real-time reporting system provides better event data reporting and processing accuracy and resilience than existing computer systems. For example, the real-time reporting system utilizes a fallback mechanism that utilizes strategically placed fallback markers to track the writing, publishing, and processing status of event data. Furthermore, in various implementations, the real-time reporting system uses different markers and stores the fallback markers in different types of memory and locations to ensure that event data is recoverable in the event of an edge device failure.

[0032] As described above, the real-time reporting system provides increased flexibility over existing computer systems. For example, in various implementations, the real-time reporting system is capable of running on most edge devices, including memory-constrained edge devices. The real-time reporting system is also device-independent and is not limited to a particular type of device or operating system. In addition, the real-time reporting system is scalable to any number of devices and can work with networks of any size (e.g., from memory-constrained devices to highly scalable cloud services). Additionally, the real-time reporting system is capable of utilizing persistent storage (e.g., key-value storage) on edge devices, cloud computing systems, or both.

[0033] As an additional example, the real-time reporting system can also work when disconnected from the corresponding cloud computing system and / or the public Internet. In fact, in various implementations, the real-time reporting system operates on an edge device that is intentionally or unintentionally disconnected from the corresponding cloud computer system. In addition to being able to operate independently without a cloud computing system, the real-time reporting system also flexibly provides customizable event data aggregation and processing.

[0034] As shown in the foregoing discussion, the present disclosure uses a variety of terms to describe the features and advantages of one or more implementations described herein. For example, the term "edge computing device" (or "edge device") refers to a computing device that acts as an intermediary between a local computing device in a network (e.g., a local network) and a computing system outside the local network (e.g., a cloud computing system). In various implementations, edge devices follow an edge computing model that physically migrates computing resources from a central data center or a public cloud to a location closer to the local device. In one or more implementations, the edge device includes a server device. For example, in an implementation, the edge device is a stack edge device that acts as a local cloud storage gateway to enable data transfer to a cloud device. In an alternative implementation, the edge device is another type of computing device. In various implementations, the edge device is implemented in hardware. In an example implementation, the edge device includes software operations.

[0035] As used herein, the term "local network" refers to a network that operates independently of the public Internet and includes edge devices. For example, a local network includes a local computing device (or "local device") and one or more edge devices. In some implementations, the local devices include Internet of Things (IoT) devices, appliances, and / or other computing devices. Additionally, in various implementations, the local network includes a computing device (using hardware or software) that provides network functionality. In various implementations, the local network connects the edge device to one or more cloud computing systems directly or via the public Internet. For example, although a local network may operate independently of the public Internet, in different cases, the local network utilizes the public Internet to provide data to computing devices located outside the local network (e.g., the operation of the local network may be enhanced based on external communications). Additionally, in some instances, the local network is also referred to as an edge network.

[0036] As used herein, the term "event data" refers to data corresponding to computing events (e.g., computing actions) that are provided by or used for services on a local network or computing devices. For example, a computing device or service within a local network generates or issues event data in the form of event data indicators. For example, event data issued by a service includes event data indicators such as central processing unit (CPU), storage, power supply, network, and bandwidth statistics and usage. In some implementations, event data also includes other event data indicators / data recorded in a service log or database log from a local device. In various implementations, event data provides the status of devices within the network and / or the status of the entire network. As described herein, event data is written, published, read, aggregated, and / or otherwise processed.

[0037] It should be noted that while portions of this disclosure describe collecting, publishing, processing, and reporting metrics for ease of explanation, these descriptions are equally applicable to broader categories of collecting, publishing, and processing other event data. For example, the event data collector described herein may be an event data collector that performs similar functions on various types of event data or entire event data.

[0038] As used herein, the term "event graph" refers to a data structure for storing event data including indicators. In various implementations, the event graph stores data in a time series format. In some implementations, the event graph saves the event data to be monitored (e.g., indicators). Examples of event graphs include concurrent hash graphs. Event graphs include writable unsealed event graphs and non-writable sealed event graphs. In some implementations, multiple event graphs are combined to form an event graph group (or simply referred to as an "event group"). In one or more implementations, the event graph includes unpublished and / or unprocessed event data. In alternative implementations, the event graph includes aggregated and processed event data, such as a consumer event graph (e.g., a "consumer event graph" includes aggregated processed data generated by a consumer component). Additionally, in some implementations, the event graph is associated with an identifier, such as an event graph identifier or an event group identifier. In many implementations, these identifiers are unique.

[0039] As used herein, the term "persistent storage" refers to a storage device that retains data after the device is powered off. In various implementations, persistent storage includes non-volatile storage. Additionally, in various implementations, persistent storage refers to storage volumes in a container associated with a stateful application (such as a database). In these implementations, persistent storage retains data beyond the lifecycle of a single container.

[0040] Similarly, the term "memory" refers to a storage element (e.g., a non-physical component such as a queue) or device that retains data while power is applied to the device. A memory element does not retain data when power is cut or interrupted. In various implementations, the memory storage element or device includes volatile memory or non-persistent memory. In various implementations, the real-time reporting system utilizes a memory queue to temporarily store event data to be published.

[0041] As used herein, the term "real-time" refers to processing that occurs immediately when events and data are generated and / or provided. In various implementations, real-time includes near real-time, which takes into account minimal processing constraints. Additionally, real-time is more current than near real-time. For example, real-time does not include long processing delays caused by processors having idle cycles or waiting for data to accumulate before processing.

[0042] Additional details about the components and elements of the real-time reporting system will now be provided. For example, Figure 1A-1B A schematic diagram of an example environment 100 (e.g., a digital media system environment) for implementing a real-time reporting system 106 is shown. In particular, Figure 1A Including environment 100, Figure 1B Additional details regarding the components and elements of the real-time reporting system 106 are provided.

[0043] like Figure 1A As shown, environment 100 includes an internal network 102, which includes various computing devices. For example, internal network 102 is a type of local network that includes edge devices 104, local devices 108, and administrator devices 110. In addition, environment 100 also includes a cloud computing system 114 connected to internal network 102 via network 118. Additional details about these and other computing devices will be described below. Fig.10 In addition, Fig.10 Additional details regarding networks are also provided, such as network 118 as shown.

[0044] As shown, the edge device 104 includes a real-time reporting system 106. In various implementations, the real-time reporting system 106 identifies, collects, receives, monitors, provides, publishes, modifies, processes, aggregates, and stores metrics (e.g., event data metrics) from local devices 108 in the enterprise internal network 102. In addition, in some implementations, the real-time reporting system 106 also provides real-time or near real-time reports of metrics from the local devices 108 to user client devices (e.g., administrator devices 110, i.e., client devices associated with administrator users). Additional details about the real-time reporting system 106 will be provided in subsequent figures below. For example, the details and components of the real-time reporting system 106 will be described below. Figure 1B Further provided in.

[0045] As described above, the enterprise internal network 102 includes local devices 108. In various implementations, the local devices 108 include computing devices that implement services and operations (e.g., IoT devices, network routing, packet processing, device reporting) within the enterprise internal network 102. For example, the local devices include IoT devices (e.g., security devices, environmental monitoring devices, or automation devices) that provide event data to the edge device 104. For example, the edge device 104 stores a service log and / or a database log that includes data received from the local device 108. In some instances, the edge device 104 sends the data (e.g., processed and aggregated data) to the cloud computing system 114.

[0046] As shown, the environment 100 includes a cloud computing system 114. Also as shown, the cloud computing system 114 includes cloud services 116. For example, the cloud services 116 provide supplemental processing for the edge device 104 in various situations. The cloud services 116 of the cloud computing system 114 also provide support for other edge devices and / or computing devices located on additional local networks. In some implementations, the cloud computing system 114 provides cloud storage for edge devices 104 and / or devices on the enterprise internal network 102.

[0047] As shown, the enterprise internal network 102 includes an administrator device 110. In one or more implementations, the administrator device 110 is a client device associated with a user (e.g., an administrator). As shown, the administrator device 110 includes a client application 112. In various implementations, the client application 112 enables an administrator or other user to interact with the real-time reporting system 106, such as providing metric queries, modifying or customizing aggregation operations, and / or viewing real-time event data reports provided by the real-time reporting system 106.

[0048] As mentioned above, Figure 1B Provides additional details about the capabilities and components of the real-time reporting system 106. By way of example, Figure 1B The edge device 104 is shown having a real-time reporting system 106 and a persistent storage 150. In some implementations, the real-time reporting system 106 is included on multiple edge devices and / or other computing devices within the enterprise internal network 102. In one or more implementations, the real-time reporting system 106 is located in whole or in part on a computing device outside the enterprise internal network 102. Additionally, the edge device 104 includes a persistent storage 150 that stores event graph data such as having processed event data (e.g., metrics). For example, the persistent storage 150 includes a consumer event graph 144.

[0049] As shown, the real-time reporting system 106 includes various components and elements, which are implemented by hardware and / or software. For example, the real-time reporting system 106 includes an event data collector 120, a monitoring agent 130, and a storage manager 134. As shown, the event data collector 120 includes an event graph manager 122, a thread publisher 124, and a rollback manager 126. Also as shown, the monitoring agent 130 includes a data processing manager 132. In addition, as shown, the storage manager 134 includes an event graph 136, a unique identifier 138, a memory queue 140, a distributed queue 142, and a rollback tag 146.

[0050] As shown, the real-time reporting system 106 includes an event data collector 120. In one or more implementations, the event data collector 120 receives, collects, identifies, modifies, provides, publishes, or otherwise manages metrics received from local devices (e.g., local devices 108). For example, the event data collector 120 utilizes the event graph manager 122 to group monitoring data into event graphs 136 (e.g., unsealed event graphs). In addition, in some instances, the event data collector 120 provides one or more event graphs (e.g., sealed event graphs) in the event graph 136 to a memory queue (e.g., temporary storage on the edge device 104) in the memory queue 140.

[0051] Additionally, in one or more implementations, the event data collector 120 publishes the event graph 136 to the monitoring agent 130 using the thread publisher 124. For example, the thread publisher 124 publishes the event graph (e.g., as published event data) to one of the distributed queues 142 associated with the monitoring agent 130. Moreover, in some implementations, the event data collector 120 (e.g., via the thread publisher 124) assigns the event graph a unique identifier in one of the unique identifiers 138 associated with publishing the event graph. As used herein, "published event data" refers to data provided by the event data collector 120 to the monitoring agent 130. In many implementations, the published event data includes an event graph read from a memory queue. Additional details regarding published event data are described below. Figure 4 Available in.

[0052] Additionally, in various implementations, the event data collector 120 utilizes the fallback manager 126 to recover lost metrics when an edge device 104 fails. For example, the fallback manager 126 strategically sets fallback markers 146, such as a current write marker to indicate a current write location of monitoring event data and / or a parser marker (e.g., a publisher marker) to indicate a current location of an event graph published to the monitoring agent 130. In various implementations, the fallback manager 126 operates or is associated with the monitoring agent 130. For example, the fallback manager 126 places a process marker to indicate a current location of an event graph 136 and / or event data being processed by the monitoring agent 130. Additional details regarding the fallback markers 146 are described below. Figure 7 Available in.

[0053] As shown, the edge device 104 includes a monitoring agent 130. In various implementations, the monitoring agent 130 facilitates receiving, modifying, identifying, aggregating, processing, and / or storing metrics published as an event graph 136. For example, the monitoring agent 130 receives a published event graph in a distributed queue in the distributed queue 142 (e.g., as published event data).

[0054] In addition, the monitoring agent 130 utilizes the backoff manager 126 to process and / or aggregate the event graph 136 removed from the distributed queue 142. In various implementations, the backoff manager 126 generates a consumer event graph 144 from the processed event data and stores the consumer event graph 144 at the persistent storage 150. Additional details regarding the functionality of the event data collector 120, the monitoring agent 130, and the persistent storage 150 will be provided in subsequent figures.

[0055] Although the event data collector 120 and the monitoring agent 130 are shown on the same component device, in some implementations, they are located on different computing devices. For example, the event data collector 120 is located on the edge device 104, while the monitoring agent 130 is located on an administrator device within the local network (or outside the local network).

[0056] Additional details regarding an example implementation of the real-time reporting system 106 are provided in Figure 2 For example, Figure 2 An example overview for implementing a real-time event data reporting system and a fallback mechanism is shown, consistent with one or more implementations. As shown, Figure 2 A series of actions 200 performed by the real-time reporting system 106 is shown.

[0057] like Figure 2 As shown, the action sequence 200 includes an action 210, namely, generating an event graph including event data on the edge device based on real-time monitoring of the event data. For example, the real-time reporting system 106 detects indicators (e.g., event data) generated by a service on a local device and provided to the edge device to be written to a service log or a database log on the edge device. Additionally, in various implementations, the real-time reporting system 106 generates an event graph from the indicators and places the event graph in a memory queue. Additional details about generating the event graph and the memory queue will be described below. Figure 3 and Figure 4 Available in.

[0058] also, Figure 2 The action sequence 200 is shown to include an action 220, namely publishing the event graph in real time to a distributed queue with a unique event group identifier. For example, the real-time reporting system 106 publishes one or more event graphs from the memory queue to the distributed queue. In some implementations, the real-time reporting system 106 groups multiple event graphs into event groups and publishes the event groups to the distributed queue. In addition, the real-time reporting system 106 generates a unique event group identifier for the event group. Additional details about publishing event graphs and unique event identifiers are described below. Figure 3 and Figure 4 Available in.

[0059] As shown, the action sequence 200 also includes an action 230, which processes the published data and persists it to a persistent storage for real-time reporting. For example, in various implementations, the real-time reporting system 106 receives event data from a distributed queue and processes the event data through one or more operations. For example, the real-time reporting system 106 provides the published data to one or more data processors (i.e., data processing services). In additional implementations, the real-time reporting system 106 provides the processed event data to one or more data aggregators to perform additional operations.

[0060] In addition, in various implementations, the real-time reporting system 106 generates a consumer event graph including processed and / or aggregated event data and provides the consumer event graph to persistent storage. In this way, the real-time reporting system 106 provides processed event data that is ready to be presented to a user client device, such as an administrator device, in the form of a real-time event data report. Additional details on processing and persisting published data are provided below. Figure 3 and Figure 4 Below Figure 8 An example of providing a report on real-time event data is provided in .

[0061] In addition, the action sequence 200 includes an action 240, in the event of a system failure, using a fallback mechanism to recover lost event data if necessary. For example, in various implementations, the real-time reporting system 106 creates and places markers to track the current progress of event data being written to the event graph, published to a distributed queue, and / or processed. Then, in the event of an edge device failure, a service restart, or other failure, the real-time reporting system 106 uses one or more markers to determine which event data has been published and which data is lost. In this way, the real-time reporting system 106 recovers any lost event data that was not published and minimizes the publication of duplicate event data. Additional details about the fallback mechanism and event data recovery will be described below. Figure 7 Available in.

[0062] As mentioned above, Figure 3 Provides more details about the operation of real-time reporting system 106. In particular, Figure 3 A block diagram of actions and operations performed by a real-time event data reporting system to improve event data reporting according to one or more implementations is shown. Figure 3 It includes a local network 300 and a remote network 310 located outside the local network and connected via one or more intermediate networks (eg, the public Internet).

[0063] As shown in the figure, Figure 3Including the edge device 104, local data source 302, computer system event record 306 and cloud storage 308 described above. As shown in the figure, the edge device 104 includes the real-time reporting system 106 described above, which has an event data collector 120, a monitoring agent 130 and a persistent storage 150. In various implementations, the local data source 302 includes a service running on a local device. In additional implementations, the local data source 302 also includes additional sources that provide event data and / or indicators.

[0064] As shown, local data source 302 emits indicators 304 (e.g., multiple event data indicators), which are captured by the computer system. Figure 3 The display indicator 304 is received at the computer system event log 306. In various implementations, the computer system event log 306 is located within the edge device 104 (represented by the small dashed line). In some implementations, the computer system event log 306 is located on another device within the local network 300.

[0065] As shown, computer system event logging 306 collects metrics 304 and provides them to cloud storage 308. As shown, cloud storage 308 is located on a remote network 310. In some implementations, cloud storage 308 is part of a cloud computing system associated with edge device 104. For example, the cloud computing system is a stack edge service cloud system that provides support for edge device 104. However, in many instances, providing metrics 304 to cloud storage 308 requires a large number of jumps, which increases the possibility of latency issues.

[0066] In some implementations, computer system event logging 306 includes a logging system for logging indicators 304. As a first example, computer system event logging 306 is a service log system for logging indicators (e.g., indicator event data) into a service log. For example, a service log is written to a file system by a system log library when the file system receives a log stream (e.g., indicator 304). In some instances, the file system includes a separate service log (or service log set) for each service or service type that provides event data. In various implementations, service log files are identified by unique names and / or service log files are based on the type or level of the service provided. Additionally, in various cases, when a first service log is full, it will rotate to a new second service log.

[0067] As another example, the computer system event record 306 is a database log. For example, a database provides a database log having multiple pages for recording changes to be applied to the database. For example, the database log acts as a "look-ahead" log, where database modifications are first written to the database log and then indexed into the database. Additionally, in various implementations, when a database log page is full, the system can rotate to another database log page to record newly received data.

[0068] As described above, in some implementations, the real-time reporting system 106 on the edge device 104 also receives the metrics 304. For example, the edge device 104 receives or creates a copy of the metrics 304 as the metrics 304 are provided to the computer system event log 306. More specifically, in many implementations, the real-time reporting system 106 embeds the event data collector 120 into the service log or database log to facilitate real-time monitoring (and replication) of the metrics 304 provided by the local data source 302. In fact, in various implementations, the event data collector 120 is a generic structure that the real-time reporting system 106 embeds into any metric recording service.

[0069] By way of example, in a first example, the real-time reporting system 106 causes the event data collector 120 to capture and convert log strings written to a file system (e.g., computer system event log 306) into event objects for addition to an event graph within the event data collector 120. In various implementations, the event data collector 120 monitors and collects a copy of the metric 304 before, after, or simultaneously with the metric 304 being written to a service log (e.g., computer system event log 306).

[0070] In a second example, real-time reporting system 106 causes event data collector 120 to collect indicators 304 in the event graph after indicators 304 are written to the database log but before they are written to the database index. In an alternative implementation, real-time reporting system 106 collects a copy of indicators 304 before or overlapping with writing indicators 304 to the database log.

[0071] Upon collecting the metrics 304, the event data collector 120 processes and / or publishes one or more event graphs and provides them to the monitoring agent 130, as described above. For example, the event data collector 120 writes the data of the event graph to a distributed queue within the monitoring agent 130. Additional details about the elements, components, and operations of the event data collector 120 are described below. Figure 4 Available in.

[0072] As described above, the edge device 104 and the real-time reporting system 106 are contained in a local network that is separate from the remote network 310. In various implementations, the real-time reporting system 106 can also operate when disconnected from the cloud storage 308, the associated cloud computing system, and / or the public network (e.g., the Internet). In fact, because the real-time reporting system 106 is locally embedded or injected into the system service of the edge device 104, and the event data collector 120, the monitoring agent 130, and the persistent storage 150 are located within the local network 300, the real-time reporting system 106 is able to provide real-time metric monitoring and reporting when the edge device 104 is in a disconnected mode (e.g., disconnected from the cloud storage 308, the associated cloud computing system, and / or the public network).

[0073] As described above, in various examples, the real-time reporting system 106 causes the monitoring agent 130 to further process the published metrics. In addition, in some examples, the monitoring agent 130 writes the processed metrics to a consumer event graph and persists the consumer event graph to a persistent store 150 (e.g., generating an event graph from the first level of the hierarchical aggregation structure). Additional details about the elements, components, and operations of the monitoring agent 130 and the persistent store 150 will be described below. Figure 5 Available in.

[0074] Go to the next picture. Figure 4 A block diagram of publishing an event graph from received event data using an event data collector within a real-time event data reporting system according to one or more implementations is shown. Figure 4 The local data source 302, the fallback manager 126, and the real-time reporting system 106 with the event data collector 120 and the monitoring agent 130 are all described above. In addition, the event data collector 120 includes an event group 402, an unsealed event graph 404, a memory queue 408 with a sealed event graph 410, and a thread publisher 412.

[0075] As described above, in various examples, the real-time reporting system 106 utilizes or enables the event data collector 120 to collect metrics 304 from the local data source 302 in real time as the data is provided or received. In various implementations, the event data collector 120 writes the metrics 304 to the event group 402. In particular, in some examples, the event data collector 120 writes the metrics 304 to the unsealed event graph 404 within the event group 402. In one or more implementations, the event data collector 120 continues to write to the unsealed event graph 404 until a trigger 406 occurs (e.g., a fullness event map threshold), at which time the event data collector 120 turns or converts the unsealed event graph 404 to one of the sealed event graphs 410 and stores it in the memory queue 408. In various examples, once an event graph is sealed, event data can no longer be written to it (e.g., a sealed event graph is a non-writable graph).

[0076] As described above, in some instances, the event data collector 120 converts the unsealed event graph 404 to one of the sealed event graphs 410 based on a trigger 406 (e.g., a full event graph threshold). In various implementations, the trigger 406 or full event graph threshold is based on reaching a file size, such as the unsealed event graph 404 becoming full, approaching a specified size, or reaching a configurable size value (e.g., a triggering event based on a file size). In some implementations, the trigger 406 or full threshold is time-based, such as sealing the unsealed event graph when a predetermined or configurable duration is reached. In one or more implementations, the trigger 406 is based on more factors, such as the number of writes or the amount of time since the last write.

[0077] In various implementations, event group 402 includes an unsealed event graph and a sealed event graph. In some instances, event group 402 includes an unsealed event graph that is being actively written and any number of sealed event graphs (including zero). For example, a new event group includes unsealed event graph 404, but does not yet have a sealed event graph 410. As described above, in some instances, an event graph is a data structure, such as a concurrent hash map, for storing indicator data that needs to be monitored.

[0078] In various implementations, event data collector 120 converts unsealed event graph 404 to one of sealed event graphs 410 by enqueuing the newly sealed event graph to memory queue 408. Additionally, when the unsealed event graph is sealed, in some instances, event data collector 120 creates a new empty event graph within event group 402. In various instances, memory queue 408 is a global single memory queue.

[0079] In various implementations, real-time reporting system 106 assigns an identifier to the event group. For example, event data collector 120 assigns a unique event group identifier to event group 402. For example, event data collector 120 generates a unique event group identifier based on a data source, service, or node providing metric 304 and / or based on other elements.

[0080] For illustration, in various implementations, the event data collector 120 generates an identifier including a prefix and a suffix for the event group. In some instances, the prefix is ​​constant, and the suffix is ​​variable (or vice versa). In one instance, the prefix indicates a local data source, a service name, and / or a node identifier associated with the indicator 304 being provided. In the case of a log file, in various instances, the prefix includes a node identifier, a service name, and / or a log file name. In the case of a database log, in some instances, the prefix includes a partition number, an event group, a log number, an epoch, and / or a log offset. Additionally, the suffix indicates a timestamp or epoch associated with writing the indicator 304 to the unsealed event graph 404, such as an epoch and / or a timestamp. Therefore, in some instances, the real-time reporting system 106 uses a unique group event identifier (e.g., a semantic identifier) ​​to quickly retrieve the required indicator or event group, including identifying the target event group from a similar event group.

[0081] By way of example, assume that service 1 is running on a local device and generates event data, such as the current memory capacity of the local device. In response to providing the current memory capacity, event data collector 120 captures these metrics and adds them to one or more event graphs in an event group. In this example, event data collector 120 generates a unique group identifier with prefix S1 based on the name of service 1. Additionally, event data collector generates a suffix T1 of the unique group identifier based on timestamp time1, which represents the time when service 1 generates the event data and / or writes it to one or more event graphs.

[0082] In various implementations, the event data collector 120 generates a monotonically increasing unique identifier. In some instances, this occurs based on a timestamp or epoch data that monotonically increases. For example, as data generated from the service 1 over a period of time is written across different event graphs and / or event groups, the suffix portion of the different generated unique identifiers continues to increase to correspond to the time at which the corresponding data was received.

[0083] like Figure 4As shown, the event data collector 120 includes a thread publisher 412. In various implementations, the thread publisher 412 publishes the event group to the monitoring agent 130. For example, in one or more implementations, the event data collector 120 uses the thread publisher 412 to dequeue the event graph from the memory queue 408, publish it to the monitoring agent 130, and iterate to the next event graph in the event group 402. In fact, in various instances, the thread publisher 412 publishes or persists the event data from the event graph to a distributed queue on the monitoring agent 130. In this way, although a high-load service can generate a large number of events, the real-time reporting system 106 uses the event graph to provide a carrier or mechanism to aggregate the event data and indicators in the memory queue to reduce the amount of data sent to the distributed queue (for example, the published event groups form another layer of the hierarchical aggregation structure).

[0084] In various implementations, the thread publisher 412 runs at given time intervals. For example, the event data collector 120 causes the thread publisher 412 to "wake up" at a given interval to read the memory queue 408. If the memory queue 408 is empty, the thread publisher 412 returns to an idle state to save computing resources. Otherwise, the thread publisher 412 quickly and efficiently publishes the queued event data. In various implementations, the interval is short enough so as not to adversely affect the real-time processing of the event data collector 120. In fact, by utilizing the memory queue 408, the real-time reporting system 106 is able to run in real time.

[0085] In some implementations, when publishing event data, the thread publisher 412 also publishes the unique identifier along with the event graph. For example, the thread publisher 412 provides the event graph and its unique event group identifier to the monitoring agent 130. As another example, the thread publisher 412 provides the event data (e.g., a subset of the event graph) and the corresponding unique identifier to the monitoring agent 130.

[0086] In various implementations, the thread publisher 412 utilizes a dump thread to publish event graph data to the monitoring agent 130. For example, the thread publisher 412 provides a snapshot of the state of one or more event graphs in the memory queue 408. As another example, the thread publisher 412 utilizes a dump thread to "dump" event data from the event graph into a common thread for more efficient publishing.

[0087] As shown, the event data collector 120 includes a fallback manager 126. Although additional details on the use of the fallback mechanism will be described below Figure 7, but a brief overview is provided here. In various implementations, the fallback manager 126 utilizes the parser to re-read event data that may be lost in write failure events (such as power loss, service restart, etc.). In these implementations, in order to promote efficiency and limit redundant processing, the parser utilizes tags that indicate event data windows that may be lost. For example, the tags include current write tags and parser tags (e.g., release tags) that indicate maximum and minimum time limits for the parser. Then, in various implementations, the event data collector 120 utilizes the fallback manager 126 to ensure that all event data is correctly analyzed and stored.

[0088] Now turn to Figure 5 , which provides additional details about monitoring agent 130. In particular, Figure 5 A block diagram is shown of persisting an organization's data to persistent storage 150 using a monitoring agent 130 within a real-time reporting system 106, according to one or more implementations. Figure 5 As shown, it includes a real-time reporting system 106 having an event data collector 120, a monitoring agent 130, and a persistent storage 150, all of which are described above. Additionally, the monitoring agent 130 includes a distributed queue 502 and a data processing service 504, the latter including a consumer 506 and an aggregator 508. In addition, Figure 5 Also included are a configurable aggregation framework 512 and validator 514 associated with the monitoring agent 130 .

[0089] As described above, the event data collector 120 publishes the event graph to the monitoring agent 130. In response, the monitoring agent 130 queues the published data on the distributed queue 502. In one or more implementations, the distributed queue 502 includes one or more physical queues distributed across multiple servers or computing devices, but they are grouped under a common name and appear as a single logical queue to the event data collector 120. In some instances, the distributed queue is a message broker or other type of data broker, including a queue that sends data between applications and services.

[0090] In various implementations, the real-time reporting system 106 utilizes event group identifiers to store published event data. For example, in one or more implementations, the monitoring agent 130 utilizes unique event group identifiers to store corresponding published event graph data. For example, in various implementations, the monitoring agent 130 utilizes unique event group identifiers as key values ​​to index published data that is persisted to the distributed queue 502. In these implementations, the distributed queue 502 is a persistent storage (e.g., data is stored in memory and on disk).

[0091] In various implementations, the monitoring agent 130 utilizes the unique event group identifier as a routing key. For example, the monitoring agent 130 utilizes the unique event group identifier or a portion thereof (e.g., a prefix or suffix) to sort and / or group incoming data and group similar event groups together. In this way, the real-time reporting system 106 is able to utilize the unique group event identifier as a semantic identifier to quickly retrieve a desired metric or event group or a group of similar event groups.

[0092] Additionally, in various examples, the monitoring agent 130 utilizes the unique event group identifier as a filter key. For example, the monitoring agent 130 utilizes the unique event group identifier to perform a duplicate check to prevent the same data (e.g., multiple identical instances of a metric received from the event data collector 120) from being processed repeatedly. In various implementations, the monitoring agent 130 utilizes the unique event group identifier to sort the characteristics of the received data, and if the data is structured in a time series format, can depict and indicate the point at which the received data has been processed.

[0093] In some implementations, the monitoring agent 130 filters out previously processed event data to prevent duplicate processing. For example, if the monitoring agent 130 has read file 2 to offset 100, if published event data between 0-100 for file 2 is received (where file 2 is identified based on its unique event group identifier), the monitoring agent 130 will filter out this data based on determining that the unique event group identifier (or a portion thereof) has been processed. In this implementation, the monitoring agent 130 utilizes the unique event group identifier to improve efficiency by ensuring that event data is only processed once (based on filtering out previously processed copies of data). In fact, in one or more implementations, the monitoring agent 130 ensures that the distributed queue 502 provides a read of event data accurately once.

[0094] In various examples, the monitoring agent 130 processes the event data stored in the distributed queue 502. For example, in various implementations, the monitoring agent 130 dequeues the data from the distributed queue 502 and processes it using the data processing service 504. As shown, the data processing service 504 includes a consumer 506 and an aggregator 508. In an example implementation, the data processing service 504 processes the data from the distributed queue in parallel to optimize the processing. Additionally, in various implementations, the data processing service 504 utilizes a unique identifier to process the event data published in the distributed queue 502.

[0095] In various implementations, the monitoring agent 130 provides the dequeued data to one or more consumers and / or one or more aggregators 508. For example, the monitoring agent 130 utilizes the aggregator 508 to perform addition, subtraction, multiplication, division, merging, transformation, and / or other operations (e.g., graph-reduce patterns or median determination or custom defined events) on the event data processed by the consumers. In this manner, the monitoring agent 130 is able to reduce the number of network calls and requests to various persistent stores (e.g., queues or persistent stores). In fact, in various implementations, the monitoring agent 130 aggregates the processed event data to reduce the number of write requests provided to the persistent store 150 (e.g., the processed event data forms another layer of the hierarchical aggregation structure).

[0096] Additionally, in various implementations, data processing service 504 (e.g., aggregator 508) creates a network graph type structure (e.g., directed acyclic graph) for the processed event data. In various implementations, real-time reporting system 106 adds one or more configurable aggregation frameworks (described below) as nodes in the network graph, which is stored in persistent storage 150.

[0097] In some instances, the monitoring agent 130 persists the processed event data to the persistent storage 150. For example, in various implementations, the monitoring agent 130 generates a consumer event graph 510 from the processed event data (e.g., data processed by the consumer 506 and aggregated by the aggregator 508) and persists it to the persistent storage 150. For example, the monitoring agent 130 generates the consumer event graph 510 based on event groups for each service instance (e.g., using unique event group identifiers).

[0098] In various implementations, the real-time reporting system 106 creates processing markers as checkpoints. In some instances, the real-time reporting system 106 utilizes processing markers to track which published event data has been read and / or processed to ensure that the event data is processed efficiently (e.g., only once). Additionally, in various instances, the real-time reporting system 106 utilizes processing markers to recover processed event data that is lost in the event of a failure associated with an edge device.

[0099] To further illustrate, in various implementations, the real-time reporting system 106 persists the data to the persistent storage 150. With respect to persisting the processed event data, in some instances, the real-time reporting system 106 also creates processing tags that indicate which data has been processed (e.g., when the data is persisted to the persistent storage 150, a checkpoint event that generates processing tags is triggered). As additional processed data is persisted to the persistent storage 150, in various instances, the real-time reporting system 106 generates and stores new processing tags. In various implementations, the processing tags indicate the current or next key value to be processed, which helps the real-time reporting system 106 process event data only once, even if the monitoring agent 130 receives and reads the same instance of event data multiple times.

[0100] In additional implementations, the real-time reporting system 106 utilizes processing tags to recover lost data. For example, in the event that the event data collector 120 and / or edge device crashes, the real-time reporting system 106 uses processing tags to determine which processed event data has been added to the consumer event graph. In this way, the processing tags act as a checkpoint (e.g., a deduplication check) to filter out duplicate events that are not accurately included in the consumer event graph.

[0101] In various implementations, the processing tag of the real-time reporting system 106 is stored in the persistent storage 150 along with the corresponding processed event data. Additionally, in some implementations, the processing tag indicates the last unique identifier corresponding to the data most recently added to the persistent storage 150 (e.g., "up to the last unique identifier corresponding to the data most recently added to the persistent storage 150"). In fact, when persisting the processed event data (e.g., raw data) to the persistent storage 150 (e.g., a key-value store), in some instances, the monitoring agent 130 also persists the unique identifier (e.g., processing tag) for the event data.

[0102] As described above, in various implementations, the persistent storage 150 is edge-based and implemented on an edge device implementing the real-time reporting system 106. In some implementations, the persistent storage 150 is implemented within the same local network as the edge device. In various implementations, the persistent storage 150 is cloud-based and implemented on a cloud computing system remote from the local system.

[0103] As shown in the figure, Figure 5Also included is a configurable aggregation framework 512. Since the amount of event data at a given time may be very large, users generally do not need to monitor and report every piece of event data in real time. Therefore, based on the configurable aggregation framework 512, in one or more implementations, the real-time reporting system 106 filters out unnecessary indicators and structures these indicators in a way that enables the real-time reporting system 106 to process the indicators of interest more efficiently and quickly.

[0104] In various implementations, the real-time reporting system 106 allows a user to configure and / or define how the monitoring agent 130 processes the published event data. For example, the real-time reporting system 106 stores a modifiable schema registry in the persistent storage 150, which the system utilizes to determine how to process the published event data pulled from the distributed queue 502. In an exemplary implementation, the configurable aggregation framework 512 is maintained in JSON, XML, or other file types.

[0105] In various implementations, the configurable aggregation framework 512 indicates threshold conditions corresponding to operations and functions performed by the data processing service 504. In one or more implementations, the configurable aggregation framework 512 provides instructions to the monitoring agent 130 on how to process data based on the data source (e.g., raw entry data), aggregation functions, validation schemes, and output formats of the published event data. In some implementations, the real-time reporting system 106 facilitates the execution of multiple configurable aggregation frameworks (e.g., aggregation mechanisms) on the same entry (e.g., raw segments of published data).

[0106] Further, in one or more implementations, the real-time reporting system 106 reduces latency by employing a configurable aggregation framework (e.g., automatically configurable aggregation). In various implementations, the configurable aggregation framework has a pattern registry that includes user-specified details and user-provided aggregation functions. In various implementations, the monitoring agent 130 reads raw entries in the distributed queue 502 to the consumer event graph 510, processes the raw data using one or more of the aggregation functions (if necessary), and persists the processed data to the persistent storage 150. In some implementations, the monitoring agent 130 generates a final entry after aggregation, which is persisted to the persistent storage 150.

[0107] Additionally, in various implementations, when a new metric (e.g., event graph data) is read, the monitoring agent 130 registers it in the pattern registry. In these implementations, the pattern registry provides a global registry for each service, edge device, or device group with monitoring metrics. In this way, the pattern registry indicates the dependencies between these metrics and aggregation functions. In addition, based on executing the aggregation functions indicated in the pattern registry, the monitoring agent 130 generates a graph (e.g., a directed acyclic graph) in which the aggregation functions are vertices that define data flows and dependencies.

[0108] As additional information, in various implementations, the monitoring agent 130 utilizes a periodic scanner to list the original entries in the distributed queue 502 and passes them to the aggregator 508 to manage the aggregated job graph list (e.g., the created graph). For example, the monitoring agent 130 generates intermediate keys after aggregation and persists the intermediate keys to the persistent storage 150. The monitoring agent 130 then reads the intermediate keys from the persistent storage 150 and / or uses the intermediate keys as input for the next aggregated set in the job graph. In various implementations, the monitoring agent 130 processes tasks in batches and / or runs them in parallel under the constraints of the service thread pool. In an exemplary implementation, the monitoring agent 130 passes the input key / value to the aggregator 508 to form the final key value. In the case where one or more aggregators in the aggregator 508 utilize multi-node (e.g., multi-stage) jobs, the entries processed from the intermediate stages are further passed to the next aggregated job when necessary.

[0109] Further, in various implementations, in one or more states of processing and aggregating event data, monitoring agent 130 removes raw entries and persists intermediate entries to persistent storage 150. In an exemplary implementation, monitoring agent 130 ensures that entries in consumer event graph 510 are associated with a time series format.

[0110] like Figure 5 As shown, a validator 514 is also included. In various implementations, the validator 514 determines when one or more conditions are met and responsively provides a corresponding indication. For example, in one or more implementations, the validator 514 identifies one or more validation rules (e.g., policies) that are validated against the received indicators. When the indicator triggers one of the validation rules, the validator 514 generates an indication that the validation rule has been met.

[0111] For example, as an example, the validation rule includes generating an alarm when the storage capacity of the edge device reaches 80%. For example, the validator 514 registers a listener to monitor whether a specific validation rule is met. When the threshold is met, the validator 514 generates an alarm. In some instances, the validator 514 publishes the alarm to a distributed queue, and the data processing service 504 recognizes the alarm when dequeuing and processing data.

[0112] In various implementations, the validator 514 is an aggregation-based validator that checks whether validation rules are satisfied on or after the aggregator 508. The validator 514 then generates and publishes an alert to the distributed queue 502, which is detected (e.g., "listened") by one or more consumers 506. In response to detecting the alert, the distributed queue provides an alert (or another indication) to the user client device (e.g., via an electronic message or a graphical user interface pop-up).

[0113] In some implementations, the real-time reporting system 106 provides an interactive interface for users to add, modify, and / or remove validation rules. For example, the validation rules are configurable and relate to how to monitor indicators and / or provide alerts to user client devices. In this way, the real-time reporting system 106 facilitates the detection of compliance-related conditions. In addition, in some implementations, the validation rules indicate the actions to be performed when the validation rules are met. Further, in various implementations, the real-time reporting system 106 adds the validation rules to a pattern registry and / or as nodes of a directed acyclic graph, which is a type of network graph.

[0114] As shown in the figure, Figure 5 Also included is a retention policy 516 associated with the persistent storage 150. In various implementations, the real-time reporting system 106 generates a retention policy 516 regarding retaining, storing, and / or removing data. For example, the retention policy 516 indicates a configurable time to remove stored data, and the real-time reporting system 106 scans the data in the persistent storage 150 (e.g., periodically) and determines whether the corresponding tag or timestamp of the entry exceeds the retention policy (e.g., checks the staleness of the entry). If so, the real-time reporting system 106 removes the data from the table or database and / or archives the data. In addition, in various examples, the real-time reporting system 106 removes stale tagged entries.

[0115] In some implementations, retention policy 516 is added to the pattern registry. For example, retention policy 516 is added as a node of a directed acyclic graph. In addition, in various implementations, retention policy 516 is located on persistent storage 150, as shown. In some implementations, retention policy 516 is located on an edge device or otherwise located within an edge-based local network. Additionally, in some implementations, while scanning data in persistent storage 150, real-time reporting system 106 removes data in persistent storage 150 when determining that a certain indicator is non-compliant.

[0116] Now let’s turn to the next picture. Figure 6 A sequence diagram for establishing and utilizing a backoff flag to prevent edge device failures is shown, consistent with one or more implementations. As shown, Figure 6 A series of actions 600 are included that are performed by the real-time reporting system 106 (eg, the event data collector 120 and the monitoring agent 130 ) with respect to the local data source 302 and the persistent storage 150 .

[0117] As shown, the sequence of actions 600 includes an action of receiving event data 602. For example, the event data collector 120 receives the event data from one of the local data sources 302, as described above. In response, as described above, in various implementations, the event data collector 120 writes an unsealed event graph, as shown in action 604. Furthermore, as described above, the event data collector 120 converts the event graph to a sealed event graph, as shown in action 606. For example, when the event graph triggers a threshold condition, the event data collector 120 seals the event graph and adds it to a memory queue.

[0118] As shown, the action sequence 600 also includes an action 608 in which the event data collector 120 generates a current write mark. For example, when the service log rotates to a new unsealed service or a database log rotation occurs, the event data collector 120 generates the current write mark. In this way, the current write mark indicates the current service log or log page of the data that has been written.

[0119] In some implementations, the current write mark indicates the current location of data being written to an active and / or unsealed event graph. In various implementations, the current write mark is persisted in persistent storage 150. For example, the current write mark indicates the name (e.g., file or log name or identifier) ​​of the event data to which the event graph has been written. In this way, the real-time reporting system 106 maintains bookkeeping information that indicates the current location of the memory queue to which received event data is being written.

[0120] For example, for a single-node system or non-partitioned database, the current write mark indicates that the node is writing to log J1 to accurately depict where the write is occurring. In other cases, the current write mark indicates when the write is moved to a different log page, such as log J3, etc. Then, when the event data collector 120 collects event information, it knows where the data is being written and sends updates to the persistent storage 150 periodically (e.g., when a rotation occurs).

[0121] As shown, the action sequence 600 includes an action 610 of publishing an event graph. For example, the event data collector 120 publishes the event graph from the memory queue to a distributed queue belonging to the monitoring agent 130, as previously described. In connection with publishing the event graph, the event data collector 120 generates a parser marker, as shown in action 612. In this way, the parser marker acts as a publishing marker or indicator, indicating at which point the event data has been published to the distributed queue. In effect, the parser marker acts as a checkpoint, indicating the latest offset position of the published event data.

[0122] As shown, the sequence of actions 600 includes an action 614 of processing and storing consumer event graph data. For example, as described above, the monitoring agent 130 processes event data from a published event graph (e.g., as published event data) and stores it in the persistent storage 150. In connection with persisting the processed event graph data, the real-time reporting system 106 generates a processing tag. For example, the monitoring agent 130 generates a processing tag, as shown in action 616.

[0123] In various implementations, the processing mark is indicated on the monitoring agent 130 until the point at which the data has been published. As shown below, the real-time reporting system 106 utilizes the processing mark as a redundant fallback mechanism to recover lost data. Additionally, in some instances, the monitoring agent 130 utilizes the processing mark as a deduplication check to ensure that data is not reprocessed.

[0124] As shown, the sequence of actions 600 includes an action 618 of detecting a failure. For example, the real-time reporting system 106 detects a failure of an edge device, a service restart, and / or a failure elsewhere in the environment. In some instances, the failure causes the event data stored in the memory queue to be lost. Therefore, the real-time reporting system 106 cannot process the data without first recovering the data. Fortunately, as described below, the real-time reporting system 106 uses a tag to implement a fallback mechanism to recover the lost data in such an event.

[0125] By way of example, the sequence of actions includes an action 620 in which the real-time reporting system 106 uses the current write mark and the parser mark to recover the lost data. In various implementations, upon detecting a failure that caused the event data to be lost, the real-time reporting system 106 initiates a general parser of the event data collector 120 to recover the lost data. In some implementations, the parser runs at a fixed interval per service and / or per database partition to discover deltas (e.g., changes) that were not published by the event data collector 120 due to the failure.

[0126] In various implementations, the parser parses the event graph (e.g., a service log / database partition) to identify a parser mark (e.g., a minimum limit in time) indicating a point at which event data has been published. The event data collector 120 then uses the parser to read from that point. In addition, the event data collector 120 identifies a current write mark indicating the last event graph to be written (e.g., a maximum limit in time). In this way, the real-time reporting system 106 performs data filtering on the event data collector 120 (e.g., the publisher) to filter out previously published data and prevent it from being republished to the monitoring agent 130, which effectively prevents duplicate processing on the monitoring agent 130 (rather than performing filtering on the monitoring agent 130, which is less efficient).

[0127] To further illustrate, the general parser mechanism implemented by the real-time reporting system 106 is a fallback mechanism that parses event data entries on the event data collector 120 to ensure that no events are missed in the event of a detected failure. Therefore, the real-time reporting system 106 utilizes a current write mark and a parser mark. As described above, the current write mark defines a maximum limit in time, meaning the furthest point in time at which parsing occurs in a time instance. This is because, in various instances, the current write mark is generated after the thread publisher 412 and indicates the last set of event graphs (e.g., service logs or database logs) to be written for publication (but may not have been published yet). Additionally, when event data is lost due to a detected failure, this indicates that the event data may be lost in a memory queue.

[0128] In addition, the real-time reporting system 106 identifies the parser tag. As described above, the parser tag stays behind the release of the event data collector 120 and indicates the released event data. In this way, the parser tag maintains the current position of the released event data, up to the current offset. Therefore, in many implementations, the parser tag instructs the parser to identify the minimum starting point (e.g., the most recent position in time) of the lost event data. Therefore, in various implementations, the current write tag and the parser tag together form a window of the minimum starting point (e.g., the parser tag) and the maximum point (e.g., the current write tag), and the parser traverses this window to recover the lost event data.

[0129] In some implementations, in recovering lost event data, the parser checks the stored file to determine if there is a current write mark. If there is no current write mark, the parser obtains the last raw entry published for the file, which includes the offset of the event data that needs to be published. Additionally, the parser parses the event data from the offset to the end of the file, which sometimes includes publishing the event data to a distributed queue for further processing, as described above. In some instances, once the file is processed, the parser updates a parser checkpoint marker that can quickly find the location where the parser should start from if a process crash occurs or where the next set of iterations should start.

[0130] As described above, the real-time reporting system 106 provides real-time capabilities to monitor and report incoming event data (e.g., created through in-memory processing). In some implementations that implement a fallback mechanism, periodic addition of markers and consideration of missing data introduces slight delays, resulting in near real-time performance. In either case, when event data is published for aggregation, it is available for immediate consumption, which reduces processing delays.

[0131] As shown, sequence of actions 600 includes an action 622 of providing a processing tag for lost data recovery. In some instances, real-time reporting system 106 requires additional information and / or assistance to recover lost data. Therefore, real-time reporting system 106 utilizes a processing tag located on monitoring agent 130 and indicating which point of the published event data has been received. For example, monitoring agent 130 provides a processing tag to event data collector 120 to indicate the last received segment of the published event data.

[0132] For example, for a file including 0-1000 event data, if there is some event data missing between the location of the processing marker and the location where the published event data occurs (e.g., the processing marker is at offset 200, and the last published location is at offset 300), in various instances, the real-time reporting system 106 provides the event data collector 120 with a data offset of 200 to republish the data starting from offset 200. In this example, the real-time reporting system 106 may perform some duplicate processing of the event data at the distributed queue when reading data events from offsets 200-300, but the data processing service detects and filters out these types of data events to prevent duplicate processing of the event data (e.g., the data is processed only once).

[0133] In various implementations, such as in the case of memory-constrained edge devices (e.g., memory-constrained edge devices), the real-time reporting system 106 adds persistent storage including a log structured merge tree (e.g., LSM) to facilitate lightweight writing and reading of event data as part of the event data publishing process. In these implementations, the LSM includes a log, a memory table, and a tree structure (e.g., a B+ tree structure). Additionally, in these implementations, the persistent storage includes write- and read-optimized storage that stores event data in a time series format, and event data from an event graph is persisted to the persistent storage along with a unique event group identifier. In additional implementations, the real-time reporting system 106 further aggregates this event data to create a final entry that is used for consumption by an administrator or user.

[0134] Additionally, in some implementations, for edge devices that are in disconnected mode, the real-time reporting system 106 utilizes the above-described framework to obtain monitoring information. For example, aggregated data written to log files is persisted to a distributed queue, and instead of sending the raw data, the aggregated data is sent to a cloud monitoring service for further processing. In this way, the real-time reporting system 106 provides a multi-level aggregation mechanism while providing users with access to monitoring information in a consumable format when the edge device is disconnected. In addition, multi-level aggregation helps optimize aggregation because less data is sent to the cloud for further processing.

[0135] Figure 7 A block diagram of using virtual edge device groups to prevent edge device loss according to one or more implementations is shown. In various implementations, the real-time reporting system 106 uses virtual edge device groups to provide simplified monitoring by grouping edge devices in the same network together as a group to be monitored. For example, Figure 7 A virtual edge device group 700 is shown including a first edge device 702a, a second edge device 702b, and a third edge device 702c. The virtual edge device group 700 may include fewer or more edge devices.

[0136] As described above, in one or more implementations, the real-time reporting system 106 forms a virtual edge device group from multiple devices (but not clustered). For example, each floor of a multi-story building forms a group to monitor data usage on each floor. As described below, the virtual edge device group provides high availability, fault tolerance, and durability by grouping multiple edge devices into a single virtual unit.

[0137] Additionally, in various instances, each of the edge devices includes multiple persistent storages, including a local KV (key-value) storage and one or more remote KV storages. For example, the first edge device 702a includes a first local KV storage 704a and a first remote KV storage 706a, the second edge device 702b includes a second local KV storage 704b and a second remote KV storage 706b, and the third edge device 702c includes a third local KV storage 704c and a third remote KV storage 706c.

[0138] In one or more implementations, within the virtual edge device group 700, the real-time reporting system 106 designates one of the devices as the primary edge device. When event data is provided to the virtual edge device group 700 from a data source, one of the edge devices (e.g., the primary device) processes the data and persists it to its local KV storage. In various implementations, the primary edge device also persists the data to the remote KV storage of the secondary device.

[0139] For example, the real-time reporting system 106 writes all events corresponding to the first edge device 702a to the first local KV storage 704a. Additionally, the real-time reporting system 106 copies the processed event data to the second remote KV storage 706b and the first remote KV storage 706a. In fact, when writing and persisting event data locally, the real-time reporting system 106 stores redundant copies of the event data remotely but also within the same local network. In this way, if an edge device in the virtual edge device group crashes and loses data, the real-time reporting system 106 provides redundant protection. In addition, redundant event data backup provides high availability when one or more edge devices in the edge device are disconnected from the cloud.

[0140] As described above, in various implementations, when creating a virtual edge device group (virtual group), the real-time reporting system 106 creates different persistent storage in each edge device. As described above, in some implementations, the persistent storage is a LSM-based structure.

[0141] In one or more implementations, the real-time reporting system 106 writes event data to an event graph stored in a memory queue (e.g., a memory table). In some instances, the real-time reporting system 106 creates and stores a replicator thread that copies a local KV storage log to a remote KV storage on other devices in the virtual edge device group. For example, each device that receives a replicated KV storage log replays the replicated KV storage log to persist the data in its remote KV storage. In one or more instances, in order to recover data, the real-time reporting system 106 replays data from its local KV storage log from the remote KV storage of the secondary edge device in sequence.

[0142] Figure 8 A graphical user interface for displaying an event data indicator report based on persistent processed event data according to one or more implementations is shown. As shown in the figure, Figure 8 A computing device 800 with a graphical user interface 802 is included, showing an event data metrics dashboard 804. In various implementations, the real-time reporting system 106 provides the event data metrics dashboard 804 in response to a request to view event data metrics for devices in the local network. In some implementations, the real-time reporting system 106 provides real-time updates of the event data metrics dashboard 804.

[0143] As shown, the event data indicator dashboard 804 includes real-time indicator reports for one or more indicators. Although some event data indicator reports are shown, in some instances, the event data indicator dashboard 804 includes additional, fewer, or different reports. For example, the event data indicator dashboard 804 includes the current capacity of the edge device.

[0144] In various implementations, the event data indicator dashboard 804 is an interactive interface that facilitates user interaction. For example, the event data indicator dashboard 804 enables a user to select a report to view more detailed information about the report. In some implementations, the event data indicator dashboard 804 provides a search function. In certain implementations, the event data indicator dashboard 804 includes interactive elements and an interface for users to interact with a configurable aggregation framework. In one or more implementations, the event data indicator dashboard 804 includes any number of interactive interface elements.

[0145] Now turn to Fig. 9 , which illustrates an example flow chart including a series of actions 900 for utilizing the real-time reporting system 106 according to one or more implementations. In particular, Fig. 9 An example sequence of actions is shown for persisting monitored and processed event data to persistent storage in accordance with one or more implementations. Fig. 9 Actions are shown according to one or more implementations, but alternative implementations may omit, add, reorder, and / or modify any of the actions shown. Fig. 9 The actions of may be performed as part of a method (e.g., a computer-implemented method). Alternatively, a non-transitory computer-readable medium may include instructions that, when executed by one or more processors, cause a computing device to perform Fig. 9 In a further implementation, a system (eg, a processing system including a processor) may perform Fig. 9 action.

[0146] As shown, action sequence 900 includes an action 910 of storing real-time event data in an event graph. For example, in an example implementation, action 910 involves real-time detection of event data to be written to an event record, the event data is generated by a data source, and a copy of the event data is stored in real time in a non-persistent memory in an event graph belonging to an event group. In various implementations, action 910 includes writing event data to an unsealed event graph. In some implementations, action 910 includes turning an unsealed event graph into a sealed event graph within an event group when a complete event graph threshold is met, the sealed event graph is a non-writable graph, and the sealed event graph is stored in a memory queue within a non-persistent memory. In various implementations, the data source is located in a local network in which an edge device is also located. In various implementations, the local network is isolated from the public network.

[0147] As further shown, the action sequence 900 includes an action 920 of publishing an event graph to a distributed queue in real time. For example, in an example implementation, the action 920 involves publishing an event graph with a unique identifier to a distributed queue as published event data in real time. In various implementations, the published event data is published to the distributed queue using a unique event group identifier as a key value. In some implementations, the unique event group identifier includes a fixed prefix based on the data source and a variable suffix including a timestamp based on the time when the event data was generated.

[0148] As further shown, the sequence of actions 900 includes an action of processing the published event data 930. For example, in an example implementation, the action 930 involves processing the published event data based on the unique identifier to generate processed event data. In one or more implementations, the action 930 includes utilizing a data processing service (e.g., a consumer and a data aggregator) to generate the processed event data.

[0149] As further shown, action sequence 900 includes an action 940 of persisting the processed event data to a persistent storage. For example, in an example implementation, action 940 involves persisting the processed event data to a consumer event graph within a persistent storage. In some implementations, the processed event data is persisted to the persistent storage using a unique identifier as a key. In additional implementations, the consumer event graph is a key-value store, and the consumer event graph uses a fixed prefix of the unique identifier as a key to store the processed event data.

[0150] In example implementations, the persistent storage is located within an edge device in the same local network as the data source (e.g., a local device in the local network providing the event data). In one or more implementations, act 940 includes providing the consumer event graph to a locally connected edge device that is disconnected from a public internet connection. In some implementations, act 940 includes providing the consumer event graph to a cloud computing system located outside the local network.

[0151] As further shown, sequence of actions 900 includes providing processed event data for display action 950. For example, in an example implementation, action 950 involves providing processed event data from a consumer event graph for display.

[0152] In some implementations, the action sequence 900 includes additional actions. For example, in some implementations, the action sequence 900 includes an action of generating a current write mark in response to rotating an unsealed event graph for a sealed event graph. In some instances, the current write mark is stored on persistent storage. In various implementations, the action sequence 900 also includes an operation of generating a parser mark in response to publishing the event graph to a distributed queue. In some instances, the parser mark indicates that the event graph has been published to the distributed queue. In some implementations, the action sequence 900 also includes determining that a memory failure in a non-persistent memory has caused event data to be lost, and recovering the lost event data based on parsing the event data copies from the current write mark to the parser mark.

[0153] In various implementations, action sequence 900 includes actions of detecting in real time event data to be written to a file system service log, which event data is generated by a service, or detecting in real time event data from a service; storing in real time a copy of event data in an event graph belonging to an event group in a memory queue, wherein storing the event data occurs before writing the event data to the service log, or storing in real time event data in an event graph belonging to an event group in a memory queue in a memory queue by writing the event data to an unsealed event graph; publishing an event graph with a unique event group identifier to a distributed queue as published event data in real time; processing the published event data based on the unique event group identifier to generate processed event data; generating aggregated processed event data by aggregating the processed event data; persisting the aggregated processed event data to a consumer event graph within a persistent storage; and providing the aggregated processed event data from the consumer event graph for display.

[0154] In one or more implementations, the action sequence 900 includes an action for detecting event data to be written to a service log by embedding an event data collector into the service. In some implementations, the action sequence 900 also includes storing the event data within an event group using the event data collector, storing the event data within an unsealed event graph within the event group, and maintaining a memory queue including one or more sealed event graphs having copies of previously received event data generated by the service. In various implementations, the action sequence 900 includes publishing the event data using the event data collector by dequeuing the event data from a memory queue, and writing the event data dequeued from the memory queue to a distributed queue based on a unique event group identifier.

[0155] In some implementations, action sequence 900 includes detecting event data to be written to a file system database log in real time, the event data being generated by one or more services, or detecting event data from a service in real time; storing a copy of event data in an event group including an event graph in a non-persistent memory in real time, wherein storing the event data occurs after the event data is written to the database log and before the event data is written to a database index of the corresponding database log; publishing the event graph with a unique identifier to a distributed queue as published event data in real time; processing the published event data based on the unique identifier to generate processed event data; and persisting the processed event data to a consumer event graph in a persistent storage.

[0156] A "computer network" (hereinafter referred to as "network") is defined as one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or another communications connection (whether wired, wireless, or a combination of wired or wireless), the computer properly views the connection as a transmission medium. The transmission medium may include networks and / or data links that can be used to carry the required program code means in the form of computer executable instructions or data structures that can be accessed by general or special purpose computers. Combinations of the above should also be included within the scope of computer-readable media.

[0157] In addition, the networks described herein may represent a network or combination of networks (e.g., the Internet, a corporate intranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a cellular network, a wide area network (WAN), a metropolitan area network (MAN), or a combination of two or more such networks) through which one or more computing devices can access the real-time reporting system 106. In practice, the networks described herein may include one or more networks that transmit data using one or more communication platforms or technologies. For example, the network may include the Internet or other data links that enable electronic data to be transmitted between respective client devices and components of a cloud computing system (e.g., server devices and / or virtual machines thereon).

[0158] Further, upon reaching various computer system components, program code means in the form of computer executable instructions or data structures may be automatically transferred from the transmission medium to the non-transitory computer readable storage medium (device), or vice versa. For example, computer executable instructions or data structures received over a network or data link may be buffered in a random access memory (RAM) within a network interface module (NIC) and then ultimately transferred to the computer system's RAM and / or a less volatile computer storage medium (device). Thus, it should be understood that a non-transitory computer readable storage medium (device) may be included in a computer system component that uses a transmission medium.

[0159] Computer executable instructions include, for example, instructions and data that, when executed by a processor, cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or set of functions. In some implementations, computer executable instructions are executed by a general-purpose computer, converting the general-purpose computer into a special-purpose computer that implements the elements of the present disclosure. Computer executable instructions can be, for example, binary files, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological behavior, it should be understood that the subject matter defined in the attached claims is not necessarily limited to the features or behaviors described above. Instead, the described features and behaviors are disclosed as example forms of implementing the claims.

[0160] Fig.10 Some components that may be included in computer system 1000 are shown. Computer system 1000 can be used to implement various computing devices, components, and systems described herein. As used herein, a "computing device" refers to an electronic component that performs a set of operations based on a set of programming instructions. Computing devices include electronic component groups, client devices, server devices, etc.

[0161] In various implementations, computer system 1000 may represent one or more client devices, server devices, or other computing devices described above. For example, computer system 1000 may refer to various types of network devices, cloud computing systems, or other systems that can access data on a network. For example, a client device may refer to a mobile device, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a tablet computer, a laptop computer, or a wearable computing device (e.g., a headset or a smart watch). A client device may also refer to a non-mobile device, such as a desktop computer, a server node (e.g., from another cloud computing system), or other non-portable device.

[0162] Computer system 1000 includes processor 1001. Processor 1001 may be a general-purpose single-chip or multi-chip microprocessor (e.g., Advanced Reduced Instruction Set Computer (RISC) Machine (ARM)), a special-purpose microprocessor (e.g., a digital signal processor (DSP)), a microcontroller, a programmable gate array, etc. Processor 1001 may be referred to as a central processing unit (CPU). Although Fig.10 The processor 1001 in the computer system 1000 is shown as just a single processor, but in an alternative configuration, a combination of processors (eg, an ARM and DSP) could be used.

[0163] The computer system 1000 also includes a memory 1003 in electronic communication with the processor 1001. The memory 1003 may be any electronic component capable of storing electronic information. For example, the memory 1003 may be embodied as a random access memory (RAM), a read-only memory (ROM), a magnetic disk storage medium, an optical storage medium, a flash memory device in a RAM, an on-board memory included in the processor, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, etc., including combinations thereof.

[0164] Instructions 1005 and data 1007 may be stored in memory 1003. Instructions 1005 may be executed by processor 1001 to implement some or all of the functions of the present disclosure. Execution of instructions 1005 may involve the use of data 1007 stored in memory 1003. Examples of various modules and components described herein may be partially or entirely as instructions 1005 stored in memory 1003 and executed by processor 1001. Various data examples described herein may be part of data 1007 stored in memory 1003 and used during execution of instructions 1005 by processor 1001.

[0165] The computer system 1000 may also include one or more communication interfaces 1009 for communicating with other electronic devices. The one or more communication interfaces 1009 may be based on wired communication technology, wireless communication technology, or both. Some examples of the one or more communication interfaces 1009 include a universal serial bus (USB), an Ethernet adapter, a wireless adapter that operates in accordance with the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 wireless communication protocol, Wireless communication adapter and infrared (IR) communication port.

[0166] The computer system 1000 may also include one or more input devices 1011 and one or more output devices 1013. Some examples of the one or more input devices 1011 include keyboards, mice, microphones, remote controls, buttons, joysticks, trackballs, touchpads, and light pens. Some examples of the one or more output devices 1013 include speakers and printers. A specific type of output device that is typically included in the computer system 1000 is a display device 1015. The display device 1015 used with the implementation of the present disclosure may use any suitable image projection technology, such as a liquid crystal display (LCD), a light emitting diode (LED), a gas plasma, or the like. A display controller 1017 may also be provided for converting the data 1007 stored in the memory 1003 into text, graphics, and / or moving images (as appropriate) for display on the display device 1015.

[0167] The various components of the computer system 1000 may be connected together via one or more buses, which may include a power bus, a control signal bus, a status signal bus, a data bus, etc. For the sake of clarity, Fig.10 Various buses are shown as bus system 1019.

[0168] Those skilled in the art will appreciate that the content of the present disclosure can be practiced in a network computing environment with many types of computer system configurations, including personal computers, desktop computers, notebook computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network personal computers, minicomputers, mainframe computers, mobile phones, personal digital assistants (PDAs), tablet computers, pagers, routers, switches, etc. The content disclosed herein can also be practiced in a distributed system environment, where local and remote computer systems are linked by a network (whether by a combination of wired data links, wireless data links, or wired and wireless data links), and both perform tasks. In a distributed system environment, program modules may be located in local and remote storage devices.

[0169] Minicomputers, mainframes, mobile phones, personal digital assistants (PDAs), tablet computers, pagers, routers, switches, etc. The content of the present disclosure can also be implemented in a distributed system environment, where local and remote computer systems are linked by a network (whether by a wired data link, a wireless data link, or a combination of wired and wireless data links) and both perform tasks. In a distributed system environment, program modules may be located in local and remote memory storage devices.

[0170] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Any features described as modules, components, or the like may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part via a non-transitory processor-readable storage medium, including instructions that, when executed by at least one processor, perform one or more of the methods described herein. Instructions may be organized into routines, programs, objects, components, data structures, and the like, which may perform specific tasks and / or implement specific data types, and may be combined or distributed as desired in various implementations.

[0171] Computer-readable media can be any available media that can be accessed by a general or special purpose computer system. Computer-readable media that store computer-executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Therefore, by way of example, and not limitation, implementations of the present disclosure may include at least two distinct computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0172] As used herein, non-transitory computer-readable storage media (devices) may include RAM, ROM, EEPROM, CD-ROM, solid-state drives (SSD) (e.g., RAM-based), flash memory, phase-change memory (PCM), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired program code means in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0173] The steps and / or actions of the methods described herein may be interchanged without departing from the scope of the claims. In other words, unless a specific sequence of steps or actions is required for the proper operation of the described method, the order of use of the steps and / or specific steps and / or actions may be modified without departing from the scope of the claims.

[0174] The term "determining" encompasses a wide variety of actions, and thus, "determining" may include computing, calculating, processing, deriving, investigating, looking up (e.g., looking up in a table, a database, or another data structure), ascertaining, etc. "Determining" may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. "Determining" may also include resolving, selecting, choosing, establishing, etc.

[0175] The terms "comprising," "including," and "having" are intended to be inclusive, meaning that in addition to the listed elements, there may be additional elements. Furthermore, it should be understood that reference to "one implementation" or "implementation" of the present disclosure is not meant to exclude the existence of other implementations that also contain the referenced features. For example, any element or feature described herein may be combined with any element or feature of any other implementation described herein, as long as they are compatible.

[0176] The content of the present disclosure may be embodied in other specific forms without departing from its spirit or characteristics. The described implementations should be regarded as exemplary rather than restrictive. Therefore, the scope of the present disclosure is indicated by the attached claims rather than by the above description. Changes within the meaning and equivalent range of the claims should be regarded as within their scope.

Claims

1. A computer-implemented method for real-time monitoring and reporting, the computer-implemented method comprising: storing event data received in the event graph in a non-persistent memory graph; publishing the event graph as part of an event group having a unique event group identifier to a distributed queue as published event data; generating processed event data by processing the published event data based on the unique event group identifier; persisting the processed event data to a consumer event graph within a persistent storage of an edge computing device; as well as The processed event data is provided from the consumer event graph for display. 2 . The computer-implemented method of claim 1 , wherein storing the event data comprises writing the event data to an unsealed event graph.

3. The computer-implemented method of claim 2, wherein storing the event data comprises: storing the event graph as part of the event group; converting the unsealed event graph into a sealed event graph within the event group, the sealed event graph being a non-writable graph when a complete event graph threshold is satisfied; and The sealed event graph is stored in a memory queue within the non-persistent memory.

4. The computer-implemented method of claim 3, further comprising: A current write mark is generated in response to rotating the unsealed event graph with respect to the sealed event graph, wherein the current write mark is stored on the persistent storage.

5. The computer-implemented method of claim 4, further comprising: A parser flag is generated in response to publishing the event graph to the distributed queue, wherein the parser flag indicates that the event graph has been published to the distributed queue.

6. The computer-implemented method of claim 5, further comprising: determining a memory failure in the non-persistent memory that caused a loss of the event data; as well as Lost event data is recovered based on parsing the copy of the event data between the current write marker and the parser marker.

7. The computer-implemented method of any one of claims 1-6, wherein the published event data is published to the distributed queue using the unique event group identifier as a key value.

8. The computer-implemented method of any one of claims 1-7, wherein the processed event data is persisted to the persistent storage using the unique event group identifier as a key.

9. The computer-implemented method of any one of claims 1-8, wherein the unique event group identifier comprises: a fixed prefix, the fixed prefix being based on a data source that generates the event data; as well as A variable suffix including a timestamp based on when the event data was generated.

10. The computer-implemented method of claim 9, wherein: The consumer event graph is a key-value store; and The consumer event graph stores the processed event data using the fixed prefix of the unique event group identifier as a key.

11. The computer-implemented method of claim 1, wherein generating the processed event data comprises utilizing a data processing service and a data aggregator to generate the processed event data.

12. The computer-implemented method of claim 1, wherein the persistent storage is located within a same local network as a data source that generates the event data.

13. The computer-implemented method of claim 1 , wherein persisting the processed event data to the consumer event data graph within the persistent store comprises: The consumer event graph is provided to a locally connected edge computing device disconnected from a public Internet connection.

14. The computer-implemented method of claim 1 , wherein persisting the processed event data to the consumer event graph within the persistent store comprises: The consumer event graph is provided to a cloud computing system external to the local network.

15. A computer-implemented method for real-time monitoring and reporting, the computer-implemented method comprising: Detect event data from services; storing the event data in an event graph belonging to an event group in a memory queue by writing the event data to an unsealed event graph; publishing the event graph with a unique event group identifier to a distributed queue as published event data; generating processed event data by processing the published event data based on the unique event group identifier; generating aggregated processed event data by aggregating the processed event data; persisting the aggregated processed event data to a consumer event graph within a persistent storage of an edge computing device; as well as The aggregated processed event data is provided from the consumer event graph. 16 . The computer-implemented method of claim 15 , wherein detecting the event data comprises embedding an event data collector into the service to detect the event data when the event data is received from the service.

17. The computer-implemented method of claim 16, wherein storing the event data in the event group comprises: storing, using the event data collector, the event data into the unsealed event graph within the event group; as well as The in-memory queue is maintained, the in-memory queue comprising a sealed event graph having copies of previously received event data generated by the service.

18. The computer-implemented method of claim 17, wherein publishing the event data using the event data collector comprises: dequeueing the event data from the memory queue; as well as The event data dequeued from the memory queue is written to the distributed queue based on the unique event group identifier.

19. A system for real-time monitoring and reporting, the system comprising: Processing systems; as well as a computer memory comprising instructions that, when executed by the processing system, cause the system to: Detect event data from services; storing in non-persistent storage a copy of the event data in the event group comprising the event graph; publishing the event graph with a unique event group identifier to a distributed queue as published event data; generating processed event data by processing the published event data based on the unique event group identifier; The processed event data is persisted to a consumer event graph within persistent storage of the edge computing device.

20. The system of claim 19, comprising additional instructions which, when executed by the processing system, cause the system to: transforming the event graph into a sealed event graph, the sealed event graph being an unwritable graph when a complete event graph threshold is satisfied; and The sealed event graph is stored in a memory queue within the non-persistent memory.

Images