Big data real-time monitoring system and method

Abstract

The invention relates to a big data real-time monitoring system and method. The big data real-time monitoring system comprises a data access module, a preprocessing standardization module, a streaming computation module, a rule model engine module, an alarm arrangement and disposal module, a storage query module and a visualization and operation and maintenance module. The data access module is used for collecting multi-source heterogeneous data from a message queue, a log agent or an Internet of Things protocol, and labeling an event timestamp and a source identifier for each piece of data; the preprocessing standardization module is used for performing deduplication, cleaning, desensitization and dimension table association on the collected data and outputting a unified event structure; the stream-oriented computing module is used for executing window aggregation, state updating and complex event processing under event time semantics and supporting water level management and just one-time semantic processing, high-timeliness and consistent monitoring is achieved through self-adaptive water level, double-layer windows and dynamic threshold gating fusion, versioning playback and approximate computing cost reduction are supported, and the method is suitable for large-scale popularization and application. And the system has automatic degradation and self-healing capabilities.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data real-time monitoring, in particular to a big data real-time monitoring system and method. BACKGROUND

[0002] With the continuous growth of data scale and concurrency in the fields of Internet of Things, Internet business, financial transactions, etc., real-time monitoring systems have become a key infrastructure to ensure business continuity and stability. The existing mainstream solutions are usually composed of data collection, preprocessing, stream computing, rule engine, alarm pushing and storage query, etc. By accessing multi-source data streams, window aggregation and rule matching are completed in the stream computing engine, alarms are generated and visualized. This kind of system can meet the real-time monitoring needs of business running state to a certain extent, but in complex business scenarios, large-scale data traffic and high proportion of out-of-order and late data conditions, there are still obvious defects, as follows:

[0003] 1. Processing timeliness and data consistency are difficult to balance. Most systems rely on fixed late tolerance threshold or processing time driven window mechanism, which is prone to window advance or delay closure when the proportion of out-of-order or late data is high, causing unstable statistical caliber. At the same time, the end-to-end consistency is not guaranteed, which is easy to cause repeated calculation or omission, affecting the reliability of the results

[0004] 2. The alarm accuracy is insufficient, and the root cause positioning efficiency is low. The alarm threshold is mostly static, lacking adaptability to business periodic fluctuations, and easy to trigger a large number of invalid alarms during business peak, holidays or promotion period. The rule engine and model reasoning are not effectively linked, and there is no fallback mechanism when the model confidence is insufficient. Complex event processing also rarely combines system topology or device dependency, which makes it difficult to effectively merge associated alarms, increasing the positioning difficulty.

[0005] 3. The historical reproduction and system resilience support are insufficient. The existing storage architecture usually does not manage hot and cold data hierarchically, and it is difficult to incrementally supplement late data or corrected data, limiting the development of historical playback and result audit. All data is uniformly entered into the precise computing and reasoning link, with high computing cost and processing delay. The system lacks service level objective (SLO) monitoring and automatic degradation mechanism, and is prone to performance decline or even interruption under high load or local failure conditions. SUMMARY

[0006] The present application aims to overcome the deficiencies of the prior art, adapt to the actual needs, and provide a big data real-time monitoring system and method to solve the current processing timeliness and data consistency difficult to balance, most systems rely on fixed late tolerance threshold or processing time driven window mechanism, when the out-of-order or late data proportion is high, the window is easy to advance or delay closing, causing unstable statistical caliber; at the same time, the end-to-end consistency guarantee is insufficient, and repeated calculation or omission is easy to occur, affecting the reliability of the result; the alarm accuracy is insufficient, the root cause positioning efficiency is low, the alarm threshold is mostly static, lacking adaptability to business periodic fluctuations, and a large number of invalid alarms are easy to trigger during business peak, holidays or promotion period; the rule engine and model reasoning do not form effective linkage, and lack of fallback mechanism when the model confidence is insufficient; complex event processing also rarely combines system topology or device dependency, resulting in that associated alarms cannot be effectively merged, increasing the positioning difficulty; historical reproduction and system resilience support is insufficient, the existing storage architecture usually does not perform hierarchical management on hot and cold data, and late data or correction data is difficult to conveniently supplement incrementally, and historical playback and result audit are limited; all data enters the accurate calculation and reasoning link, and the calculation cost and processing delay are high; the system lacks service level objective (SLO) monitoring and automatic degradation mechanism, and is easy to appear performance decline or even interruption under high load or local failure conditions.

[0007] In order to achieve the purpose of the present application, the technical solution adopted by the present application is as follows: a big data real-time monitoring system is designed, which comprises a data access module, a preprocessing standardization module, a stream computing module, a rule model engine module, an alarm arrangement and disposal module, a storage query module, and a visualization and operation and maintenance module;

[0008] The data access module is used for collecting multi-source heterogeneous data from a message queue, a log agent or an Internet of Things protocol, and marking an event timestamp and a source identifier for each data;

[0009] The preprocessing standardization module is used for de-duplication, cleaning, desensitization and table association of the collected data, and outputs a unified event structure;

[0010] The stream computing module is used for executing window aggregation, state updating and complex event processing under event time semantics, and supports water level management and exactly-once processing semantics;

[0011] The rule model engine module is used for executing threshold rule judgment and online model reasoning on the aggregation result, and generating a primary alarm;

[0012] The alarm arrangement and disposal module is used for merging, suppressing and silencing the primary alarm, and triggering automatic disposal according to a strategy;

[0013] The storage query module is configured to write the details, the aggregation, and the alarm record into a time sequence storage and a versioned data lake respectively, and support historical playback and audit;

[0014] The visualization and operation module is configured to show real-time indicators, alarm situation, and platform health status.

[0015] Preferably, the stream computing module comprises an adaptive waterline and a double-layer window unit, which dynamically adjusts the waterline according to the delay distribution, and maintains two types of windows, i.e., fast windows and stable windows, in parallel, the former is used for low-latency alarm output, and the latter is used for late data correction and caliber consistency calibration.

[0016] Preferably, the rule model engine module comprises a dynamic threshold generation unit, which automatically calculates the threshold based on trend-seasonality decomposition and quantile confidence interval, and adaptively adjusts the alarm level according to the uncertainty.

[0017] Preferably, the rule model engine module adopts a gating fusion strategy, when the model output exceeds the safety interval defined by the rule or the model confidence is lower than the threshold, it automatically falls back to rule judgment, and triggers distribution drift monitoring to prompt model retraining.

[0018] Preferably, the complex event processing of the stream computing module is based on event time execution sequence and pattern matching, and combines service / network / device dependency topology for topology-aware detection, to improve the identification accuracy of root cause related events.

[0019] Preferably, the alarm arrangement and disposition module introduces a dependency graph driven root cause clustering, de-duplicates and merges multi-dimensional symptom alarms, and supports P0-P3 hierarchical suppression, silencing, and escalation strategies.

[0020] Preferably, the storage query module realizes incremental backfill of late and corrected data through the versioned data lake, supports repeatable historical playback according to event time, and realizes exactly-once write based on two-phase commit.

[0021] Preferably, the preprocessing standardization module comprises an approximate statistics unit, which uses an error-controllable sketch structure for preliminary judgment screening, triggers accurate calculation and model inference after hitting, to reduce overall computing cost and processing latency.

[0022] Preferably, the visualization and operation module sets service level objectives SLO of the monitoring platform itself, including P99 alarm delay, processing error rate, and state growth rate, and automatically degrades to rule-only or fast window-only mode when SLO fails; the alarm arrangement and disposition module further configures a self-healing disposition arrangement, which performs fuse, capacity expansion, rollback, or restart according to the disposition manual, and automatically rolls back to a safe state when disposition fails.

[0023] The application also provides a big data real-time monitoring method, comprising the following steps:

[0024] S1, continuously acquiring multi-source streaming data by a data access module and marking an event timestamp;

[0025] S2, performing deduplication, cleaning, desensitization and table association to generate a unified event;

[0026] S3, adopting an adaptive water line and a double-layer window strategy to perform window aggregation, state updating and mode matching under an event time semantic;

[0027] S4, performing dynamic threshold determination and online model reasoning by a rule and model engine, generating a primary alarm based on a gate fusion and falling back to a rule when confidence is insufficient;

[0028] S5, performing root cause clustering in combination with a dependency graph, performing deduplication, merging, suppression and muting on the primary alarm and forming a disposal plan;

[0029] S6, triggering multi-channel notification and self-healing action according to the disposal plan and recording disposal results;

[0030] S7, writing details, aggregations and alarms into a time sequence storage and a versioned data lake respectively and ensuring exactly-once writing through a two-phase commit;

[0031] S8, performing repeatable playback on historical data according to an event time, performing A / B evaluation and threshold re-labeling on rules and models and backfilling corrected data in an incremental manner.

[0032] Compared with the prior art, the application has the beneficial effects that:

[0033] 1. The application combines real-time results and delayed correction results in combination with a two-phase commit and versioned data storage, realizes consistency of statistical indicators before and after and traceability while ensuring timeliness of output.

[0034] 2. The application introduces a dynamic threshold method of trend and seasonal decomposition in combination with a gate fusion strategy of rules and models, automatically falls back to rule determination when a model is unstable, reduces a false alarm rate, analyzes a complex event processing in combination with a system topology and cooperates with a root cause clustering algorithm, effectively reduces repeated alarms and shortens fault positioning time.

[0035] 3. The application supports versioned incremental backfilling of late and corrected data through data storage, facilitates repeatable playback of historical data and index caliber auditing, increases an approximate calculation screening step in a data preprocessing stage, reduces overall calculation amount and response delay, sets a key SLO indicator and has an automatic degradation and self-healing strategy, and still can keep continuous operation of core functions under high load or abnormal conditions. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the system of the present invention;

[0037] Figure 2 This is a schematic diagram of the streaming computing module of the present invention;

[0038] Figure 3 This is a schematic diagram of the rule model engine module of the present invention;

[0039] Figure 4 This is a schematic diagram of the preprocessing standardization module of the present invention;

[0040] Figure 5 This is a schematic diagram of the monitoring method of the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0042] A real-time big data monitoring system and method, see [link to relevant documentation] Figures 1 to 5 It includes a data access module, a preprocessing standardization module, a streaming computing module, a rule model engine module, an alarm orchestration and handling module, a storage and query module, and a visualization and operation and maintenance module;

[0043] The data access module is used to collect multi-source heterogeneous data from message queues, log brokers or IoT protocols, and to label each data item with an event timestamp and source identifier;

[0044] The preprocessing standardization module is used to deduplicate, clean, de-identify, and associate the collected data with dimension tables, and output a unified event structure.

[0045] The streaming computing module is used to perform window aggregation, state updates and complex event processing under event-time semantics, and supports water level management and exactly-once processing semantics;

[0046] The rule model engine module is used to perform threshold rule determination and online model inference on the aggregation results, and generate primary alarms;

[0047] The alarm orchestration and handling module is used to merge, suppress, and silence primary alarms, and trigger automated handling according to policies;

[0048] The storage query module is used to write detailed, aggregated, and alarm records to time-series storage and versioned data lake respectively, and supports historical playback and auditing;

[0049] The visualization and operations module is used to display real-time metrics, alarm status, and platform health status.

[0050] Specifically, the stream computing module includes an adaptive waterline and a double-layer window unit, dynamically adjusts the waterline according to a delay distribution, and maintains two types of windows, namely, a fast window and a stable window in parallel, the former is used for low-latency alarm output, and the latter is used for late data correction and caliber consistency calibration.

[0051] More specifically, the rule model engine module includes a dynamic threshold generation unit, automatically calculates a threshold based on trend-seasonality decomposition and quantile confidence interval, and adaptively adjusts an alarm level according to uncertainty.

[0052] Further, the rule model engine module adopts a gating fusion strategy, automatically falls back to rule determination when model output exceeds a safety interval defined by the rule or model confidence is lower than a threshold, and triggers distribution drift monitoring to prompt model retraining.

[0053] Still further, the stream computing module performs sequence and pattern matching based on event time, and performs topology-aware detection based on service / network / device dependency topology to improve the identification accuracy of root cause related events.

[0054] It is worth noting that the alarm arrangement and disposal module introduces a dependency graph driven root cause clustering, de-duplicates and merges multi-dimensional symptom alarms, and supports P0-P3 hierarchical suppression, silencing and escalation strategies.

[0055] It is worth noting that the storage query module realizes incremental backfill of late and corrected data through a versioned data lake, supports repeatable historical playback according to event time, and realizes exactly-once write based on two-phase commit.

[0056] It is worth introducing that the preprocessing standardization module includes an approximate statistics unit, which uses an error-controllable sketch structure for preliminary judgment and screening, triggers accurate calculation and model reasoning after hitting, to reduce the overall computing cost and processing delay.

[0057] It is worth emphasizing that the visualization and operation module sets service level objectives SLO of the monitoring platform itself, including P99 alarm delay, processing error rate and state growth rate, and automatically degrades to only rule or only fast window mode when SLO fails; the alarm arrangement and disposal module also configures self-healing disposal arrangement, executes fuse, capacity expansion, rollback or restart according to the disposal manual, and automatically rolls back to a safe state when disposal fails.

[0058] A big data real-time monitoring method is also provided, including the following steps:

[0059] S1, continuously acquiring multi-source stream data and marking event timestamps by the data access module;

[0060] S2, performing de-duplication, cleaning, desensitization and table association to generate unified events;

[0061] S3, adopt adaptive water level line and double-layer window strategy to perform window aggregation, state update and pattern matching under event time semantics;

[0062] S4, perform dynamic threshold determination and online model reasoning by rule and model engine, generate primary alarm based on gated fusion and fallback to rule when confidence is insufficient;

[0063] S5, combine dependency graph to perform root cause clustering, perform deduplication, merging, suppression and silence on primary alarm and form treatment plan;

[0064] S6, trigger multi-channel notification and self-healing action according to treatment plan, and record treatment result;

[0065] S7, write details, aggregation and alarm into time series storage and versioned data lake respectively, and ensure exactly-once write through two-phase commit;

[0066] S8, replay history data according to event time, perform A / B evaluation and threshold re-calibration on rules and models, and backfill corrected data in incremental manner.

[0067] Embodiment one

[0068] Industrial production equipment running state monitoring

[0069] This embodiment is suitable for real-time monitoring of the running state of stamping, injection molding and other production equipment, including key parameters such as vibration, temperature, current, etc.

[0070] In the data access stage, real-time data at the equipment end is collected through OPC-UA or MQTT protocol, partitioned by equipment number through Kafka, and device clock time and source identification are added to each data.

[0071] In the preprocessing link, data deduplication and time correction are performed, temperature units are unified, and equipment information is associated with production line, workshop and other dimensional data.

[0072] In the streaming computing link, the system dynamically adjusts the water level line according to the distribution of the last 5 minutes delay, and at the same time maintains a 5-second fast window and a 30-second stable window. The fast window is used for rapid output of alarm, and the stable window is used for late data correction. The complex event processing is set to the case that vibration RMS continuously rises and temperature rising rate exceeds threshold within 120 seconds.

[0073] In the rule and model evaluation stage, dynamic threshold is generated combined with shift and seasonal characteristics, and confidence is calculated through online anomaly detection model. When the model output confidence is low or the delay is out of limit, the system falls back to rule determination.

[0074] The alarm orchestration module is based on the relationship diagram between equipment, production line, and power supply circuit. It merges relevant alarms and triggers measures such as speed reduction, shutdown warning, and work order dispatch.

[0075] Detailed data and aggregation results are stored in ClickHouse, while late and corrected data are written to the Iceberg table for version management. The system supports replaying historical data by event time for review. The SLO is set to P99 alarm delay of no more than 5 seconds. When the limit is exceeded, the system automatically degrades to rule judgment and quick window calculation only, and restarts the job or switches to a backup partition through a self-healing mechanism.

[0076] Example 2

[0077] E-commerce promotional activity transaction chain monitoring

[0078] This embodiment is used to monitor key indicators such as PV, UV, order rate, and payment success rate during large-scale promotional events.

[0079] The system collects Nginx access logs, transaction service and inventory service event tracking data, and payment callback information through a log collection component, and uses logical business time uniformly.

[0080] In the preprocessing stage, information such as users, product categories, and channels are linked in dimension tables, blacklisted users are removed, and fields involving privacy are anonymized.

[0081] The streaming computing uses a 10-second fast window and a 1-minute stable window. CEP detects whether there are any breakpoints or timeouts in the conversion of adding to cart, placing an order, and making payment in the link.

[0082] Dynamic threshold calculation is combined with holiday and event calendar adjustments to the baseline. An online logistic regression model is used to predict abnormal risks. The gating mechanism reverts to rule-based judgment when the model is unstable.

[0083] The alarm orchestration module merges alarms upstream based on service dependencies (such as Gateway, Cart, Order, Pay), and handles them by issuing rate limiting policies, circuit breaking some functions, and rolling back inventory tokens.

[0084] Aggregated data is written to columnar storage, while raw and corrected data are entered into Iceberg for version management. It supports replaying and reviewing by active window and optimizing thresholds. SLO sets the P99 alarm delay to no more than 3 seconds. When the limit is exceeded, the alarm merging threshold is automatically increased and only the quick window calculation is retained.

[0085] Example 3

[0086] Real-time risk control of bank transactions

[0087] This embodiment focuses on real-time risk identification for bank card and account transactions.

[0088] System access transaction flow, device fingerprint, geographic location and customer history portrait data, Kafka enables idempotent write to prevent duplication.

[0089] The preprocessing stage includes ID unification, geographic location correction, outlier truncation, and sensitive information desensitization.

[0090] Stream computing uses 2-second fast window and 20-second stable window, and CEP is used to capture device replacement, geographic location abnormal transition, and large transaction composite events.

[0091] Dynamic threshold is generated by customer group (such as business travel, local, international), model uses gradient boosting tree online inference, drift monitoring triggers retraining suggestion through PSI and KS index; When the model result is unreliable, the system falls back to rule judgment and increases the risk level.

[0092] Root cause clustering is merged according to the association of accounts, devices, IPs and merchants, and the disposal includes secondary verification, limit, delay of entry and list synchronization.

[0093] Data storage uses two-phase commit to ensure consistency, Iceberg version snapshot facilitates audit and regulatory traceability, historical data can be replayed according to event time, SLO requires P99 judgment delay not more than 1 second, and when exceeding, the system only retains whitelist, blacklist and hierarchical limit rules.

[0094] Embodiment four

[0095] Urban traffic operation monitoring

[0096] This embodiment is suitable for monitoring the traffic flow and congestion of urban road intersections.

[0097] The target count data of roadside units and cameras is transmitted through MQTT, and preliminary aggregation is performed through edge nodes.

[0098] The preprocessing stage includes error correction of camera counting and association of road event tables (such as holidays, construction information).

[0099] Stream computing sets 5-second fast window and 60-second stable window, first uses KLL sketch to approximate calculation of quantile as pre-screening, and then performs detailed calculation and model inference after hitting the exception; CEP is used to detect abnormal congestion and adjacent road section backflow events.

[0100] Dynamic threshold adjusts according to morning and evening peak and seasonal changes, model outputs congestion probability and gives uncertainty, and gating mechanism guarantees judgment stability.

[0101] Alarm arrangement and disposal combine road topology and signal timing information, disposal measures include phase timing adjustment, induction screen prompt and dispatch of on-site disposal work order.

[0102] Columnar storage supports second-level multi-dimensional drilling, and Iceberg version management supports historical review. The SLO requirement from alarm generation to handling triggering takes no more than 10 seconds, and fixed time protection policy is enabled when the limit is exceeded.

[0103] Embodiment Five

[0104] Cloud-native microservice availability monitoring

[0105] This embodiment is used for monitoring the availability, response time and error rate of microservices in a Kubernetes cluster.

[0106] Data access includes application instrumentation, Envoy access logs and Pod event information.

[0107] The preprocessing stage merges information such as services, versions, namespaces, etc.

[0108] Stream computing uses a 15-second fast window and a 1-minute stable window, and topology-aware CEP identifies the increase in downstream error rate caused by upstream anomalies based on service call chains.

[0109] Dynamic thresholds are generated by version and environment (production, testing), and the model uses a time series prediction residual method. The gating mechanism falls back to the SLO threshold judgment when the model is abnormal.

[0110] Root cause clustering merges alarms to upstream services or shared dependencies (such as databases and caches), and handling includes automatic scaling, rolling restart, gray rollback and circuit breaking.

[0111] Detailed indicators are written to ClickHouse, events and configurations are written to Iceberg, and historical playback is facilitated by version and indicators. SLO includes control targets for interface latency and error rate, and only fast window calculation and error rate hard threshold are retained when the limit is exceeded.

[0112] Embodiment Six

[0113] Cross-region multi-active architecture operation monitoring

[0114] This embodiment is used for traffic and health monitoring of multi-active systems across provinces or clouds.

[0115] Edge access clusters in each region write data to local Kafka and generate a globally unique ID for each event.

[0116] The preprocessing stage performs regional time difference correction and link identification completion.

[0117] Stream computing maintains an adaptive water line in each region, and a longer stable window (2 minutes) is used in the convergence layer for consistency calibration.

[0118] The dynamic threshold is generated by region, season, and holiday. The model is used to assess whether to trigger cross-region traffic migration.

[0119] The root cause clustering combines multiple hierarchical topologies such as regions, lines, machine rooms, and clusters. The treatment measures include traffic redistribution, read-only mode switching, and DNS weight adjustment.

[0120] The Iceberg table in each region is bidirectionally replicated and retains version snapshots, supporting cross-region consistency audit and playback. The SLO requires that the delay of cross-region alarm aggregation be no more than 8 seconds. When the limit is exceeded, only local alarms and treatments are performed, and data is backfilled after link recovery.

[0121] Comparative Example 1

[0122] Only processing time and static threshold

[0123] This scheme only uses a fixed-length processing time window (1 minute) for statistics, has no event time and late processing mechanism, and relies entirely on fixed thresholds or simple year-on-year calculations. The storage uses a single engine detail table and has no version management.

[0124] When the proportion of late or out-of-order data rises, the window is closed prematurely or delayed, resulting in unstable statistical results. Late data cannot be backfilled and is discarded or repeatedly counted. Promotional peak periods trigger a large number of invalid alarms, increasing the burden on manual processing, and root cause positioning is slow.

[0125] Comparative Example 2

[0126] Event time but fixed late threshold, single window, no rollback mechanism

[0127] This scheme uses event time in calculations, but the late threshold is fixed at 5 seconds, and only a single sliding window is maintained. Although a model inference is introduced, there is no gate rollback and drift monitoring, and the storage end also does not support versioned data backfill.

[0128] In different business stages, the fixed late threshold cannot adapt to actual delay changes, resulting in fluctuations in statistical scope. The model is prone to distribution drift during holidays or business mode switching, leading to false positives, and lacks rollback protection. Late data cannot be incrementally supplemented, and historical data is difficult to replay or audit.

[0129] The specific comparison data is as follows:

[0130]

[0131]

[0132]

[0133] Summary

[0134] Compared with the comparative example scheme, the application introduces adaptive water level and double-layer window mechanism on the basis of event time processing, can effectively correct the late data while maintaining the second-level response, avoid statistical caliber fluctuation, through dynamic threshold, gated fusion and topology-aware complex event processing, realizes the accurate triggering of alarm and the rapid positioning of root cause, reduces the invalid alarm rate, at the same time, adopts versioned incremental backfill and SLO driven automatic degradation, self-healing mechanism, so that the system can still maintain consistent output and stable operation under high load or abnormal conditions, while the comparative example has obvious deficiencies in consistency, alarm quality and operation resilience.

[0135] In addition, the components designed by the application are all general standard components or components known to those skilled in the art, the structure and principle of which can be known by technical personnel through technical manual or through conventional experimental method, and those skilled in the art can fully realize it without unnecessary description, and the content protected by the application does not involve improvement of internal structure and method.

Claims

1. A big data real-time monitoring system, characterized in that, The system comprises a data access module, a preprocessing standardization module, a stream computing module, a rule model engine module, an alarm arrangement and disposal module, a storage query module, and a visualization and operation module. The data access module is configured to collect multi-source heterogeneous data from a message queue, a log agent, or an Internet of Things protocol, and mark each piece of data with an event timestamp and a source identifier. The preprocessing standardization module is configured to remove duplicates, clean, desensitize, and associate the collected data with a table, and output a unified event structure. The stream computing module is configured to perform window aggregation, state updating, and complex event processing under the event time semantics, and support water level management and exactly-once processing semantics. The rule model engine module is configured to perform threshold rule judgment and online model reasoning on the aggregation results, and generate primary alarms. The alarm arrangement and disposal module is configured to merge, suppress, and mute the primary alarms, and trigger automatic disposal according to a strategy. The storage query module is configured to write detailed, aggregated, and alarm records into a time series storage and a versioned data lake, and support historical playback and auditing. The visualization and operation module is configured to display real-time indicators, alarm situations, and platform health states.

2. The big data real-time monitoring system of claim 1, wherein, The stream computing module comprises an adaptive water level and a double-layer window unit, dynamically adjusts the water level according to the delay distribution, and maintains two types of windows, namely fast windows and stable windows in parallel, the former is used for low-latency alarm output, and the latter is used for late data correction and caliber consistency calibration.

3. The big data real-time monitoring system of claim 1, wherein, The rule model engine module comprises a dynamic threshold generation unit, automatically calculates the threshold based on trend-seasonality decomposition and quantile confidence interval, and adaptively adjusts the alarm level according to the uncertainty.

4. The big data real-time monitoring system of claim 1, wherein, The rule model engine module adopts a gating fusion strategy, automatically falls back to rule judgment when the model output exceeds the safety interval defined by the rule or the model confidence is lower than the threshold, and triggers distribution drift monitoring to prompt model retraining.

5. The big data real-time monitoring system of claim 1, wherein, The complex event processing of the stream computing module is based on event time execution sequence and pattern matching, and combines service / network / device dependency topology for topology-aware detection, to improve the identification accuracy of root cause related events.

6. The big data real-time monitoring system of claim 1, wherein, The alarm arrangement and disposal module introduces a dependency graph driven root cause clustering, removes and merges multi-dimensional symptom alarms, and supports P0-P3 hierarchical suppression, muting, and escalation strategies.

7. The big data real-time monitoring system of claim 1, wherein, The storage query module realizes incremental backfilling of late and corrected data through a versioned data lake, supports repeatable historical playback according to event time, and realizes exactly-once writing based on two-phase commit.

8. The big data real-time monitoring system of claim 1, wherein, The preprocessing standardization module comprises an approximate statistics unit, uses an error-controllable sketch structure for preliminary judgment screening, triggers accurate calculation and model reasoning after a hit, to reduce the overall computing cost and processing delay.

9. The big data real-time monitoring system of claim 1, wherein, The visualization and operation module sets service level objectives (SLO) of the monitoring platform itself, including P99 alarm delay, processing error rate, and state growth rate, and automatically degrades to a rule-only or fast window-only mode when the SLO is violated; the alarm arrangement and disposal module is also configured with a self-healing disposal arrangement, performs fuse, capacity expansion, rollback, or restart according to a disposal manual, and automatically rolls back to a safe state when the disposal fails.

10. A big data real-time monitoring method, applicable to the big data real-time monitoring system of any one of claims 1-9, characterized in that, The method comprises the following steps: S1, continuously acquiring multi-source streaming data by the data access module and marking event timestamps; S2, performing deduplication, cleaning, desensitization, and table association to generate unified events; S3, using adaptive water line and double-layer window strategy to perform window aggregation, state update, and pattern matching under event time semantics; S4, performing dynamic threshold determination and online model inference by the rule and model engine, generating primary alarms based on gated fusion, and falling back to rules when confidence is insufficient; S5, performing root cause clustering by combining dependency graphs, performing deduplication, merging, suppression, and silencing on primary alarms, and forming a handling plan; S6, triggering multi-channel notifications and self-healing actions according to the handling plan, and recording the handling results; S7, writing details, aggregations, and alarms into time-series storage and versioned data lake respectively, and ensuring exactly-once write through two-phase commit; S8, replaying historical data according to event time, A / B evaluating and threshold re-calibrating rules and models, and backfilling corrected data in incremental manner.

Images