A real-time monitoring and early warning method for operation data based on stream computing

Abstract

The application relates to the technical field of data processing, and discloses an operation data real-time monitoring and early warning method based on stream computing, which comprises the following steps: collecting physical operation indexes to determine a current operation state, encapsulating the operation state and control permission into a state token, and delivering the state token to a downstream node; when a time window reaches a boundary, verifying token consistency to control window closure; isolating and storing normal and delayed arrival data, and respectively aggregating the data into summaries; extracting the summaries to perform non-recalculation algebraic synthesis to generate index results, and performing coverage update on historical records; routing the results to a tree-shaped baseline according to token permission to perform update or isolated deliberation to output early warning results. The application decouples the strong dependence of a time window on a delayed data stream, avoids memory expansion caused by full recalculation through memory isolation aggregation, and effectively distinguishes system fluctuation and business migration based on tree-shaped baseline evolution, thereby improving the accuracy of early warning results.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method for real-time monitoring and early warning of operational data based on streaming computing. Background Technology

[0002] Stream processing systems typically utilize event time watermarking mechanisms to trigger the closure of time windows in real-time business data computation. When the stream processing link encounters network fluctuations or sudden backpressure on computing nodes, the transmission delay of some data streams can hinder the global event time watermarking process, causing downstream nodes to remain in a waiting state for extended periods and resulting in the suspension of local computation tasks. When processing delayed, out-of-order data, existing stream computing logic often relies on reading historical detailed data from the underlying layer for full recalculation. This process consumes physical memory space, leading to a degradation in the processing performance of operators.

[0003] Furthermore, when using the metrics derived from stream processing for operational monitoring, conventional monitoring systems often employ fixed static baseline models, lacking the ability to perceive the system's operational state and distinguish business evolution trends. In the face of unstable business environments, static baseline models struggle to determine whether data deviations are due to normal business evolution or abnormal system fluctuations, leading to false alarms in the output monitoring and early warning results.

[0004] Therefore, this invention proposes a real-time monitoring and early warning method for operational data based on streaming computing to address the shortcomings of existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a real-time monitoring and early warning method for operational data based on streaming computing. This method solves the problems that existing streaming processing systems are prone to causing computing tasks to be suspended and memory to swell when processing delayed data, as well as the difficulty of distinguishing between normal business evolution and abnormal fluctuations caused by static monitoring baselines, resulting in false alarms.

[0006] To address the above problems, the present invention provides the following technical solution:

[0007] The first aspect of this invention provides a method for real-time monitoring and early warning of operational data based on streaming computing, comprising the following steps:

[0008] The system collects physical operation indicators from the data stream access point of the data stream processing system, determines the current operating state of the system based on the evolution trend of the physical operation indicators, encapsulates the operating state and control permissions into causal state tokens, and delivers them to the specified downstream operators according to the directed acyclic graph dependency relationship of the computation topology.

[0009] When the downstream time window reaches the event time boundary, the consistency characteristics of the causal state tokens received within the specified scope are checked, and the closure confirmation of the time window is controlled accordingly. At the same time, the data that arrives normally and the data that arrives late within the time window are isolated in the underlying memory and aggregated into independent digest structures respectively.

[0010] Extract the isolated storage summary structure, perform recalculation-free algebraic alignment synthesis to generate the final adjudication result, and perform overwrite update or branch conflict convergence on the historical records in the downstream early warning indicator storage medium based on the final adjudication result and feature matching rules.

[0011] Based on the access permissions carried by the causal state token, the final decision result is routed to the tree-like baseline structure to perform incremental updates or isolated reviews. The topological branches of the baseline nodes are dynamically adjusted according to the structural offset characteristics after review, and the target baseline parameters are extracted to output the early warning result.

[0012] Furthermore, physical operation metrics include message queue lag, computation topology backpressure ratio, and data output throughput. The current operating state of the system is determined based on the evolution trend of physical operation metrics, including: after horizontally aligning physical operation metrics of the same period, calculating the time first derivative of the data output throughput to quantify the rate of throughput performance evolution; and combining a state machine model configured with a dual-threshold hysteresis interval and a derivative descent threshold to determine the system operating state as a stable state, a benign burst state, a malignant blocking state, or a transitional recovery state.

[0013] Furthermore, the causal state token is encapsulated by the epoch signature, scope signature, runtime parameters, window closure permission bit, and baseline admission permission bit; the targeted delivery to the specified downstream operator according to the directed acyclic graph dependency relationship of the computation topology includes: mapping the scope signature to the actual physical computation slot, querying the dependency mapping relationship table to obtain the list of associated nodes, and directly pushing the causal state token to the storage memory where the target downstream operator node is located.

[0014] Furthermore, verifying the consistency characteristics of causal state tokens and controlling the closure confirmation of the time window includes: forcibly switching to a waiting state when the time window reaches the event time boundary; extracting all expected causal state tokens and performing a global logical AND operation on the window closure permission bit carried by the causal state tokens; and controlling the time window to switch from the waiting state to the closure confirmation state to trigger settlement when the logical AND operation result is true.

[0015] Furthermore, the closing confirmation of the control time window includes physical clock degradation logic: when the time window enters the waiting state, the physical wall clock timer is started synchronously; when the waiting delay of the timer reaches the timeout tolerance threshold and the global logic AND operation is not completed, the missing permission state is ignored, and the time window is forcibly pushed to the closing confirmation state; during the life cycle of this time window, the baseline access permission position of the data corresponding to the missing token is invalidated and a degradation label is attached.

[0016] Furthermore, isolating and aggregating normally arriving data and delayed arriving data in the underlying memory includes: allocating independent adjudication storage areas and residual storage areas in the underlying physical memory; generating adjudication summary structures for data arriving before the time window closes in the adjudication storage area; and removing the original multidimensional structure from data arriving delayed after the time window closes, extracting its core numerical fields and calling algebraic aggregation operators according to the corresponding epoch signature to perform incremental folding updates, and maintaining fixed-dimensional residual incremental summaries.

[0017] Furthermore, the residual storage area is configured with memory management logic: based on the available memory capacity of the current node, the fixed byte size of a single residual digest, and the engineering safety factor, the maximum epoch lifetime parameter is dynamically calculated; by comparing the actual residence time of the residual incremental digest through a background asynchronous thread, the memory space occupied by the digest whose actual residence time is greater than or equal to the maximum epoch lifetime parameter is forcibly released, and the delayed data arriving outside the parameter boundary is discarded to the dead letter queue.

[0018] Furthermore, before generating the final ruling, the process includes: when the time window is in a waiting state, combining the current ruling summary, epoch signature, and computational concurrent subtask number to generate a unique state fingerprint, constructing a triplet provisional state containing the epoch signature, state fingerprint, ruling summary, and monotonically increasing logical version number, and outputting it downstream.

[0019] Furthermore, the extraction of the isolated storage digest structure and the execution of recalculation-free algebraic alignment synthesis to generate the final adjudication result include: after the time window transitions to a closed confirmation state, extracting the adjudication digest and residual digest based on the same epoch signature and performing direct algebraic operations; for indicators containing retraction update semantics, deducting the discarded scalar corresponding to the retraction operation in the residual digest and superimposing the incremental scalar corresponding to the append operation to achieve state merging.

[0020] Furthermore, based on the final ruling and feature matching rules, the historical records in the downstream early warning indicator storage medium are overwritten or branch conflict converged. This includes: using the primary key identifier, scope signature, and unique state fingerprint of the write instruction as ternary verification conditions to overwrite the historical provisional states in the downstream storage medium that support idempotent semantics; when the association break causes the verification to fail, the logical version numbers of the new and old records are compared, and an overwrite is performed based on the monotonically increasing attribute of the version number, and the original copy of the conflict is written to the dead letter queue.

[0021] Furthermore, the final decision result is routed to the tree-structured baseline for incremental updates based on the access permission carried by the causal state token. This includes: when the baseline access permission bit is determined to be valid, the data is routed to the corresponding active baseline node, and incremental iterative calculations are performed on the node mean and sum of squared deviations using the current baseline mean, the number of arriving samples, and the total number of cumulative samples to complete the baseline parameter update.

[0022] Furthermore, the isolation review and dynamic adjustment of the baseline node topology branches include: when the baseline access permission bit is invalid, suspending the current data to an isolated memory area to collect independent profile statistical features; after the review period ends, dividing the difference between the sample mean and the original baseline mean by the original baseline standard deviation with an added zero constant to obtain the structural offset; when the structural offset is greater than the preset structural offset threshold and the effective sample density is greater than the preset tolerance lower limit, confirming the substantial migration of the working condition, splitting and deriving a new baseline node with the original active baseline node as the parent node, and setting it as the current active branch.

[0023] Furthermore, it also includes asynchronous pruning and backtracking early warning logic: calculate the maximum survival time limit after an active node is transferred to the archive state, and use asynchronous threads to forcibly release the archived historical baseline memory space that exceeds the time limit; for data in the isolation review period or degraded state, extract the parameters of the nearest steady-state parent node along the tree-like baseline structure to the root, construct a temporary early warning boundary based on the standard deviation of the parent node's mean by a preset multiple, and perform anomaly fallback interception.

[0024] A second aspect of the present invention provides a real-time monitoring and early warning system for operational data based on streaming computing, comprising:

[0025] The status acquisition and routing module is used to collect physical operation indicators, determine the running state, and encapsulate them into causal state tokens for targeted delivery to the directed acyclic graph.

[0026] The consensus control and state separation module is used to decouple the time window closure logic based on causal state tokens and isolate normal data and delayed data at the underlying level, and perform dimensionality-reduced incremental folding aggregation.

[0027] The residual synthesis and overlay module is used to extract isolated summaries, perform recalculation-free algebraic alignment, output provisional and final decision results, and perform overlay updates for downstream storage based on state fingerprints and logical clocks.

[0028] The baseline evolution and early warning module is used to manage the tree-like baseline structure, use access permissions to intercept non-steady-state data for isolated review, drive the derivation and abandonment of baseline branches based on structural offsets, and output real-time early warning results.

[0029] A third aspect of the present invention provides a computer device, including a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the computer program is executed by the processor, it implements the aforementioned method for real-time monitoring and early warning of operational data based on streaming computing.

[0030] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the aforementioned method for real-time monitoring and early warning of operational data based on streaming computing.

[0031] This invention provides a method for real-time monitoring and early warning of operational data based on streaming computing. It has the following beneficial effects:

[0032] 1. This invention determines the current operating state by collecting physical operating indicators from the access point of the stream processing system, and encapsulates the operating state and control permissions into a status token, which is then directed to downstream nodes along the computing topology. This technical feature enables downstream computing nodes to obtain the objective operating status of the global link before processing business data, avoiding blindly receiving and processing data under sudden backpressure or network congestion conditions, and ensuring the operational stability of the stream processing system in complex network environments.

[0033] 2. This invention extracts the window closure permission bit from the associated node's status token to perform global logical AND operations, and combines this with the physical timer's timeout tolerance threshold to execute degradation logic, forcibly advancing the time window from a waiting state to a closed confirmation state. This scheme decouples the traditional time window from its strong dependence on unilaterally delayed data streams, solves the problem of global computation tasks being suspended due to local data transmission delays, and ensures continuous computation and output of the data stream processing link.

[0034] 3. This invention isolates and stores normally arriving data and delayed arriving data in physical memory. It aggregates the delayed arriving data and extracts the residual incremental digest, then performs a recalculation-free algebraic synthesis with the normal decision digest. This memory isolation and incremental synthesis mechanism eliminates the need for full backtracking and recalculation of underlying historical detailed data, effectively controlling the physical memory consumption of the operator when processing out-of-order data, and reducing the probability of system memory bloat and operator performance degradation.

[0035] 4. This invention intercepts and isolates non-steady-state data based on the access permissions carried by the status token. It determines whether a substantial shift in business conditions has occurred by calculating the structural offset of sample features, and then derives a new baseline node branch when the structural offset exceeds the limit. This dynamic evolution method can distinguish between normal business evolution and transient data fluctuations in the algorithm structure, solving the problem of false alarms in non-steady business environments caused by conventional static monitoring models, and improving the accuracy of the final early warning results. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the system architecture according to an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram illustrating the internal working principle of the status acquisition and routing module in an embodiment of the present invention.

[0039] Figure 4 This is a schematic diagram illustrating the internal working principle of the consensus control and state separation module in an embodiment of the present invention.

[0040] Figure 5 This is a schematic diagram illustrating the internal working principle of the residual synthesis and overlay module in an embodiment of the present invention.

[0041] Figure 6 This is a schematic diagram illustrating the internal working principle of the baseline evolution and early warning module in an embodiment of the present invention.

[0042] Figure 7 This is a schematic diagram of the end-to-end delay comparison curve under sudden back pressure according to an embodiment of the present invention;

[0043] Figure 8 This is a schematic diagram comparing the state memory usage under different hysteresis data ratios in an embodiment of the present invention;

[0044] Figure 9 This is a schematic diagram illustrating the real-time early warning effect based on dynamic tree baseline according to an embodiment of the present invention.

[0045] Among them, 10 is the state acquisition and routing module; 20 is the consensus control and state separation module; 30 is the residual synthesis and coverage module; and 40 is the baseline evolution and early warning module. Detailed Implementation

[0046] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] See attached document Figure 1 This invention provides a real-time monitoring and early warning system for operational data based on streaming computing. The system includes:

[0048] The status acquisition and routing module 10 is deployed at the data flow access end of the distributed stream processing computing topology. The status acquisition and routing module 10 collects physical operation indicators at fixed intervals and determines the current operating state of the system based on preset thresholds. The status acquisition and routing module 10 encapsulates the determination result and access control parameters into a causal status token, and delivers the causal status token to the specified downstream operator according to the directed acyclic graph dependency relationship of the stream processing computing topology.

[0049] The consensus control and state separation module 20 is deployed in the downstream time window operator nodes. When the event time reaches the boundary, the consensus control and state separation module 20 checks the consistency of the received causal state tokens and confirms the closure of the consensus state control time window. In the underlying memory, the consensus control and state separation module 20 allocates data arriving within the normal interval and data arriving after timeout to different storage areas, aggregating them to generate independent digest data structures.

[0050] The residual synthesis and overlay module 30 receives the data structure processed by the consensus control and state separation module 20. When the token consistency condition is not met, the residual synthesis and overlay module 30 generates provisional data with attached feature identifiers. After the consistency condition converges, it extracts the isolated and stored digest data, performs algebraic alignment operations, and generates the final decision result. The residual synthesis and overlay module 30 performs overlay updates on downstream historical records based on feature matching rules and controls the merging operation of parallel conflicting data branches.

[0051] The baseline evolution and early warning module 40 is deployed at the early warning output end of the stream processing computing topology. The baseline evolution and early warning module 40 manages the tree-like baseline structure, processing data admission and incremental updates of node data based on the parameters carried by the final decision result. For data in an isolated state, the baseline evolution and early warning module 40 performs compatibility statistical verification to adjust the correlation between baseline nodes and extracts real-time target baseline parameters to output early warning results.

[0052] See attached document Figure 2 This invention provides a method for real-time monitoring and early warning of operational data based on streaming computing, comprising the following steps:

[0053] S100 collects physical operation indicators of the data stream access end, determines the operating status of the stream processing system based on the operation indicators, encapsulates the determination result into a causal state token, and selectively delivers the causal state token to the specified downstream operator according to the computational topology dependency.

[0054] S200 checks the consistency characteristics of causal state tokens within the specified delivery range when the event time boundary is reached, controls the current time window to enter the verifiable state, and allocates the normal arrival data and the delayed arrival data within the time window to an independent digest storage area for isolation.

[0055] S300 outputs a provisional decision with a feature identifier when the consistency of the causal state token is not achieved. After the consistency convergence, it extracts the isolated digest, performs algebraic synthesis to generate the final decision, and performs overwrite update or branch conflict convergence on the downstream history based on the final decision.

[0056] S400 routes the processed data to the early warning baseline structure to perform incremental updates or compatibility reviews based on the admission parameters carried by the final decision, adjusts the status of the baseline storage node based on the review results, extracts the target baseline parameters, and outputs the early warning results.

[0057] To further clarify the implementation of each technical aspect of the present invention, the following will provide a detailed description of the implementation of each functional module involved above and its internal processing flow.

[0058] In this embodiment, the status acquisition and routing module 10 is deployed at the data flow access end of the stream processing computing topology, and is responsible for acquiring physical operation index data and converting the system operation environment characteristics into control parameters that flow downstream.

[0059] See attached document Figure 3 As shown, specifically, the status acquisition and routing module 10 collects the operating index data of the current computing partition according to a set fixed period. In order to comprehensively evaluate the performance bottlenecks of the entire input-output link of the stream processing system, the status acquisition and routing module 10 sets up monitoring probes at the data flow access nodes of the stream processing computing topology.

[0060] At the end of each fixed cycle, the status acquisition and routing module 10 extracts the message queue lag, computation topology backpressure ratio, and data output throughput of the current computing partition. In this embodiment, the message queue lag represents the amount of data backlog at the current data stream access point that has not been processed by the computation logic. The computation topology backpressure ratio represents the saturation level of the operator node's computing resources. The data output throughput represents the number of data entries actually processed per second by the stream processing system.

[0061] Considering that the aforementioned multi-dimensional operational metrics data originate from different physical components of the underlying message middleware and stream computing engine, to avoid timing misalignment of metrics caused by network latency, the status acquisition and routing module 10 introduces time alignment logic after acquiring the data. As a preferred approach, based on a set fixed-period sampling timestamp, a tolerance time deviation window (e.g., 50ms) is used to horizontally align the message queue lag, computation topology backpressure ratio, and data output throughput within the same acquisition batch, discarding incompletely matched malformed data slices to ensure consistency of operating conditions in subsequent status determinations.

[0062] For the probe implementation of obtaining the back pressure ratio of the computing topology and the throughput of the data output end, those skilled in the art can call the monitoring indicator application interface exposed inside the stream computing engine to obtain statistical values. The implementation logic is a well-known technology in this field and will not be described in detail here.

[0063] To capture the dynamic evolution trend of the streaming system's processing capacity and avoid one-sided judgments based solely on a single static value, the state acquisition and routing module 10 calculates the time first derivative of the data output throughput after acquiring it. This time first derivative quantifies the rate of decay or increase in system throughput performance. The formula for calculating the time first derivative is:

[0064] ;

[0065] In the formula, This represents the first derivative of the throughput calculated in the current period. This indicates the throughput of the data output terminal acquired in the current period. This indicates the throughput of the data output acquired in the previous cycle. This indicates the duration parameter of the set fixed period.

[0066] In this embodiment, Real numbers are preset to be greater than 0 to avoid calculation errors due to a denominator of 0. Additionally, during the first cycle of a system cold start or fault restart, due to the lack of... Historical data, status acquisition and routing module 10 will The default value is 0.

[0067] After acquiring the above indicators, the state acquisition and routing module 10 performs logical decision calculations for the running state in conjunction with the state machine model. In non-stationary network environments, directly triggering state switching based on a single threshold can easily lead to frequent system oscillations. Therefore, the state acquisition and routing module 10 incorporates a dual-threshold hysteresis interval model.

[0068] This dual-threshold hysteresis interval model is configured with a high back pressure threshold parameter and a low back pressure threshold parameter. A numerical range is maintained between the high and low back pressure threshold parameters to avoid frequent state switching triggered when system indicators fluctuate near a single critical point. The state acquisition and routing module 10 is also configured with a derivative descent threshold parameter characterizing a rapid decline in processing performance exceeding a threshold, and a duration condition parameter for the time dimension. As a specific implementation, the value range and specific values ​​of the aforementioned high back pressure threshold parameter, low back pressure threshold parameter, and derivative descent threshold parameter are determined based on the system's extreme throughput decay curve under offline stress testing conditions.

[0069] The status acquisition and routing module 10 presets four levels of system operating states, including stable state, benign burst state, malignant blockage state, and transitional recovery state. The status acquisition and routing module 10 triggers the switching of operating states based on the condition comparison results. When the acquired computational topology backpressure ratio is less than the low backpressure threshold parameter and the duration of this condition meets the duration period condition parameter, the status acquisition and routing module 10 determines that the stream processing system is in a stable state.

[0070] When the calculated backpressure ratio is greater than the high backpressure threshold parameter and the first derivative of the throughput in the current cycle is greater than or equal to 0, the state acquisition and routing module 10 determines that the flow processing system is in a benign burst state.

[0071] When the calculated backpressure ratio is greater than the high backpressure threshold parameter and the first derivative of the throughput in the current cycle is less than the negative value of the derivative descent threshold parameter, the state acquisition and routing module 10 determines that the flow processing system is in a severe blocking state. The derivative descent threshold parameter is a positive real number, and this condition physically characterizes the system as being under severe backpressure, with the throughput rapidly decreasing at a rate exceeding the tolerance limit.

[0072] When the calculated topology backpressure ratio falls from the malignant congestion state to below the high backpressure threshold parameter and is within the set time buffer window period, the state acquisition and routing module 10 determines that the stream processing system is in a transition recovery state. The duration of this time buffer window period is set based on the average empirical time required for the underlying message queue to empty the historical backlog data.

[0073] After completing the state determination, the state acquisition and routing module 10 generates a causal state token based on the determination result and performs targeted delivery of parameters. The state acquisition and routing module 10 establishes a data structure consisting of 5 parameters to construct the causal state token. These 5 parameters include epoch signature, scope signature, runtime parameters, window closure permission bit, and baseline admission permission bit.

[0074] The epoch signature records the time boundary identifier corresponding to the current fixed period and serves as the identifier of the current event time window. The runtime parameters record the specific system runtime state calculated in the above steps. The window closure permission bit records the permission status for triggering settlement operations for downstream time windows. The baseline access permission bit records the permission status for writing current period data and updating the downstream monitoring and statistical baseline.

[0075] Scope signatures are used to limit the scope of control exerted by causal state tokens on downstream logical operations. In the underlying implementation logic, the lower-level characteristics of the scope signature need to establish a one-to-one mapping relationship with the specific physical execution unit of the underlying data stream distribution platform. Specifically, the scope signature is mapped to a specific data partition identifier in the message middleware architecture, or a physical subtask identifier within the stream computing engine. This mapping relationship sinks the logical control scope down to the actual physical computing slots.

[0076] Finally, the state acquisition and routing module 10 queries the downstream associated operator nodes covered by the current scope signature based on the pre-established directed acyclic graph dependency mapping table in the stream processing computation topology. According to the list of dependent associated nodes, the state acquisition and routing module 10 directly delivers the generated causal state token to the storage memory where the target downstream operator node resides. By using targeted delivery to specific nodes instead of a network-wide broadcast mechanism, the state acquisition and routing module 10 can effectively block state interference between data streams corresponding to signatures from different scopes.

[0077] In this embodiment, the consensus control and state separation module 20 is deployed on the downstream node of the computing topology where the time window operator resides. This module decouples event timing from computation triggering conditions and optimizes the underlying state memory. The running states of the time window include at least a receiving state, a waiting state, and a closing confirmation state. This process can be divided into multiple execution logics.

[0078] See attached document Figure 4 As shown, in the specific implementation, the consensus control and state separation module 20 receives the causal state token and performs consensus verification based on scope consistency. In conventional stream processing architectures, the triggering of computation often strongly depends on the monotonically increasing event time watermark. However, in a vicious blocking state, severe lag in one-sided data flow can hinder the global watermark advancement, causing computation to be suspended.

[0079] Therefore, when the time watermark of the underlying stream processing engine advances to the event time boundary of the current time window, the consensus control and state separation module 20 suspends the running state of the current time window from the receiving state and forcibly switches it to the waiting state, pausing the execution of regular data settlement.

[0080] In this embodiment, the consensus control and state separation module 20 extracts all expected causal state tokens under the epoch signature corresponding to the current time window. The range of the set of expected causal state tokens is pre-defined by the dependency mapping relationship of the stream processing computation topology. The consensus control and state separation module 20 extracts the window closure permission bit carried by each causal state token in this set and performs a logical AND operation. The set of expected scope signatures is set as follows. ,in, The total number of branches depends on the topology. For any signature. The window closure permission bit carried by its causal state token is denoted as The consensus control and state separation module 20 calculates the global consensus determination criteria. The calculation formula is as follows:

[0081] ;

[0082] If and only if At that time, the consensus control and state separation module 20 determines that the global consensus condition has been met.

[0083] Once the consensus conditions are met, the consensus control and state separation module 20 controls the current time window to change from a waiting state to a closed confirmation state, triggering the actual data settlement operation. As a preferred approach, this verification mechanism decouples the strong binding relationship between event time and actual calculation triggering, avoiding skewed calculation results due to data gaps caused by severe lag in data flow on one side in downstream operator nodes within the expected range.

[0084] Considering that in a distributed system environment, a local network partition or an abnormal failure of an upstream operator node could lead to the permanent loss of causal state tokens signed in a specific scope, relying solely on the above logic and consensus verification would highly likely result in the system falling into an indefinite deadlock. To remedy this logical flaw, the consensus control and state separation module 20 introduces a physical wall clock timer that is independent of the event time evolution.

[0085] When the current time window enters a waiting state, the consensus control and state separation module 20 synchronously starts the physical clock timer. As a specific implementation, the timeout threshold parameter of the physical clock timer is set based on the maximum end-to-end latency that the business system can tolerate; for example, it can be set to 3000ms based on the historical latency statistics of the 99th percentile of the business link.

[0086] Set the initial timestamp of the physical wall clock when the current time window enters the waiting state as follows: The current physical clock is The preset timeout tolerance threshold is (For example, 3000ms). The consensus control and state separation module 20 calculates the waiting delay according to a specific detection frequency. If the following timeout trigger formula is met:

[0087] ;

[0088] and The consensus control and state separation module 20 forces the triggering of timeout degradation logic.

[0089] During the execution of the timeout degradation logic, the consensus control and state separation module 20 ignores the missing permission state and forcibly pushes the current time window directly from the waiting state to the closed confirmation state.

[0090] Simultaneously, for specific scope signatures that fail to deliver causal state tokens on time, the consensus control and state separation module 20 forcibly strips the baseline access permission bit of the corresponding data within the lifecycle of the current time window. For example, it forcibly overwrites the permission flag bit from 1 to 0 and assigns a degradation label to this part of the data. This operation allows incomplete or extremely delayed anomalous data to still participate in the numerical calculation of the current window to ensure the business continuity of stream processing, but it also loses the right to update the downstream underlying monitoring and early warning baseline model, thereby establishing an isolation barrier to prevent abnormal fluctuation data from polluting the overall baseline parameters of the system.

[0091] In time-window settlement mechanisms, the handling of late data is often the core pain point leading to state memory bloat. Even after the time window transitions to a closed confirmation state, there may still be delayed data belonging to that time window that continues to arrive in the underlying network channel due to severe congestion. If the multi-dimensional original structure attached to the aforementioned delayed data is completely stored in memory, the continuous accumulation of data will cause the underlying state memory of downstream operator nodes to overflow.

[0092] In this embodiment, the consensus control and state separation module 20 allocates two independent storage areas in the underlying physical memory: a decision storage area and a residual storage area. For normal data arriving before the time window closes, the consensus control and state separation module 20 performs regular aggregation calculations in the decision storage area to generate a decision summary structure.

[0093] For data arriving delayed after the time window has closed, the consensus control and state separation module 20 abandons retaining its original multidimensional data structure and instead extracts the core numerical fields involved in the indicator calculation from the delayed data, and performs incremental folding and aggregation according to the corresponding epoch signature. Let's say it is in the epoch... The existing state summary of the residual storage area within each epoch is as follows: The numerical vector extracted from the newly arrived hysteresis data is Then the state folding update formula is:

[0094] ;

[0095] in, This represents the algebraic aggregation operator (such as incremental summation or Max / Min extreme value replacement operator) corresponding to the business monitoring metrics. Through this aggregation operator, the incremental folding operation performed by epoch signature always maintains only a fixed-dimensional summary corresponding to the monitoring metric type. For monitoring metrics containing rollback update semantics, at least a positive incremental summary and a negative rollback summary are maintained separately, effectively suppressing the linear expansion of storage volume caused by the increase in the number of delayed data entries.

[0096] To further prevent resource exhaustion caused by the indefinite accumulation of long-tailed, delayed data, the consensus control and state separation module 20 configures a maximum epoch lifetime parameter for the residual storage area. This maximum epoch lifetime parameter represents the limit of physical time that the residual increment digest is allowed to reside in the underlying physical memory. Specifically, the consensus control and state separation module 20 dynamically calculates this maximum epoch lifetime parameter based on the memory load state of the node's underlying layer. The calculation formula is:

[0097] ;

[0098] in, The available state backend memory capacity allocated to the current operator node (e.g., the size of the RocksDB available buffer); The fixed byte size of a single residual digest; The baseline physical duration span for a single epoch; This is an engineering safety adjustment factor used to reserve a memory buffer defense line.

[0099] The consensus control and state separation module 20 periodically compares the actual residence time of the residual increment digest in memory with the maximum epoch lifetime parameter through a background asynchronous thread. When the actual residence time is greater than or equal to the maximum epoch lifetime parameter, the consensus control and state separation module 20 forcibly triggers a memory reclamation operation to clear the expired residual increment digest.

[0100] As a fallback mechanism to prevent crashes, if some extremely delayed data arrives after the maximum epoch lifespan, the consensus control and state separation module 20 will directly perform a discard operation and output it to the dead letter queue to ensure the stability of the underlying memory state during the long-term operation of the stream processing engine.

[0101] See attached document Figure 5As shown, in this embodiment, the residual synthesis and overlay module 30 is deployed at the end operator node of the data processing link. It is used to handle the recalculation-free merging of late data participating in the final result calculation and to execute the exception update fallback strategy. This process can be divided into multiple execution logics.

[0102] In a distributed stream processing environment, differences in ingestion rates from different data sources or network jitter often cause short-term delays, suspending the computation of downstream time windows. To ensure that downstream business systems can obtain periodic monitoring metrics without creating blind spots, when the time window is in a waiting state and some normally arriving data is received, the residual synthesis and overlay module 30 extracts the provisional decision summary currently in memory.

[0103] To ensure the traceability of the output state and prevent state crosstalk within the expected range, in this embodiment, the residual synthesis and overlay module 30 combines the current adjudication digest with the scope signature carried in the received causal state token and uses a hash algorithm to generate a unique state fingerprint. As a preferred approach, the scope signature not only contains node association information on the data flow path, but also explicitly encapsulates the concurrent subtask number of the physical computing node processing the data.

[0104] The residual synthesis and overlay module 30 constructs fingerprint features based on a hash mapping model. The hash mapping model for this state fingerprint is as follows:

[0105] ;

[0106] In the formula, This represents the unique state fingerprint generated. This represents a secure hash algorithm function (such as the SHA-256 algorithm). Indicates the signature of the current epoch. Indicates the concurrent subtask number, The core values ​​representing the summary of the ruling, This indicates the scope signature carried in the causal state token. This represents a string concatenation operator based on a preset delimiter. By introducing multi-dimensional features of the epoch and concurrent task space into the computation, this model can effectively avoid hash collisions between different parallel processing branches within the same time window.

[0107] After obtaining the unique state fingerprint, the residual synthesis and overlay module 30 constructs a triplet identifier output structure. This triplet identifier includes the epoch signature, the unique state fingerprint, and the current adjudication summary result, and writes a monotonically increasing logical version number for each provisional adjudication result and the final adjudication result. The residual synthesis and overlay module 30 sends this triplet identifier as a provisional state to the downstream system. This provisional state is used to provide temporary early warning indicator references to the downstream system to meet the real-time requirements of the business dashboard display.

[0108] As physical time progresses, when the consensus control and state separation module 20 determines that the global consensus condition has been met, or triggers timeout degradation logic causing the time window to enter a closed confirmation state, the residual synthesis and overlay module 30 performs an epoch-aligned residual algebra synthesis operation. It then triggers another epoch-aligned residual algebra synthesis operation when the residual storage area summary corresponding to that time window is updated. In conventional stream processing mechanisms, the arrival of late data often requires reading historical detailed data from the underlying state backend for a full recalculation, which consumes excessive computing resources and exacerbates the current backpressure on the system.

[0109] To reduce the performance overhead of full recalculation, in this embodiment, the residual synthesis and overlay module 30 extracts the normal adjudication digest in a closed state and the hysteretic residual digest stored in the residual storage area from physical memory based on the same epoch signature. The residual synthesis and overlay module 30 performs direct algebraic operations on the extracted digests. Since the residual increment digest has already been folded and aggregated in the early stage and only retains a single state scalar, this algebraic operation process only involves scalar addition and subtraction or extreme value comparison logic.

[0110] The specific execution rules for algebraic operations depend on the type semantics of the input data stream. For monitoring metrics that only involve appending writes and summation, the residual synthesis and coverage module 30 directly performs algebraic addition on the value of the adjudication digest and the value of the residual digest. For monitoring metrics that include data withdrawal and update semantics, the residual synthesis and coverage module 30 needs to scan the residual digest and perform algebraic synthesis with withdrawal semantics. The algebraic synthesis formula is as follows:

[0111] ;

[0112] In the formula, Indicates the signature of the era The final monitoring indicator values ​​synthesized below, This represents the base value of the normal adjudication summary corresponding to this era. This indicates the total number of records containing rollback operations carried in the residual summary. Indicates the first The negative rollback scalar corresponding to the reversal operation. This indicates the total number of records containing append operations carried in the residual summary. Indicates the first The positive increment scalar corresponding to the append operation.

[0113] This operational logic physically implements a recalculation-free merging process that first subtracts historical obsolete states and then adds the latest state, ensuring algebraic accuracy even when the data is in an out-of-order evolution state. Similarly, for monitoring indicators of the maximum value type, the residual synthesis and coverage module 30 compare their values ​​and take the larger one as the final synthesized value.

[0114] During algebraic synthesis, the residual synthesis and overlay module 30 synchronously checks the triggering conditions of the current synthesis operation. If, in the pre-processing stage, the time window is forcibly triggered by a degradation closure due to a physical clock timer timeout, the residual synthesis and overlay module 30 forcibly adds a degradation tag to the final result data packet after synthesis. This degradation tag is physically represented by a specific isolation flag in the data structure. Result data with this degradation tag is only used to trigger downstream normal-level alarms, but is not allowed to be written into the system statistical baseline model used for long-term evolution, thereby establishing a physical-level contamination isolation barrier.

[0115] After completing the algebraic synthesis, the residual synthesis and overlay module 30 performs scope constraint checks and conflict data convergence operations. Since a provisional state has already been sent downstream in the preceding logic, the final result obtained at this point needs to overwrite the old data downstream. To achieve this, the residual synthesis and overlay module 30 sends an update command to the downstream storage medium.

[0116] In this embodiment, the requirement for the downstream warning indicator storage medium to perform historical provisional state failure overwrite is that it possesses idempotent update semantics based on the primary key. Storage media supporting idempotent update semantics can be, for example, a Redis key-value database or a ClickHouse columnar database table with primary key characteristics. The residual synthesis and overwrite module 30 uses the epoch signature as the primary key identifier for the time window and attempts to overwrite the old provisional decision result in-situ within the storage medium by matching this primary key identifier.

[0117] To prevent erroneous overwriting of data in other computational branches under severe network disruptions, as a preferred approach, the residual synthesis and overwrite module 30 issues a ternary verification instruction before performing the overwrite operation. This ternary verification requires that the primary key identifier, scope signature, and unique state fingerprint of the current write instruction match the historical provisional state stored at the target location. Only when all ternary verification conditions are met will the residual synthesis and overwrite module 30 authorize the underlying storage medium to perform an in-situ overwrite operation.

[0118] Considering that in abnormal scenarios, the stream processing topology may experience partial crashes and restarts, leading to a break in the scope topology association of the token. In this case, the scope signature carried by the current synthesis result cannot match the historical fingerprint recorded in the downstream storage medium, resulting in the failure of the ternary verification. Faced with this association breakage condition, the residual synthesis and overlay module 30 adopts a strategy based on logical clocks to converge the data branches. Since the physical server clocks of distributed clusters often have drift deviations, simply relying on arrival time may cause the old state to overwrite the new state.

[0119] Therefore, the residual synthesis and overwrite module 30 extracts the built-in logical version number of each result record. A comparison operation is performed based on the monotonically increasing attribute of the logical version number. Only when the logical version number of the newly written record is greater than or equal to the historical record version number is the data forcibly overwritten using the last write-wins strategy, abandoning the hard comparison restriction of the old fingerprint.

[0120] Meanwhile, as a fallback mechanism to prevent data loss, the residual synthesis and overlay module 30 packages and extracts the original provisional state copies that conflict due to verification failures, along with the abnormal verification logs, and pushes them into an independent dead-letter queue using an asynchronous background thread. To prevent the dead-letter queue from continuously expanding and exhausting the disk, the background thread is configured with a rolling cleanup strategy based on data storage volume or retention days. This dead-letter queue is used to persistently retain abnormal conflict data so that developers can perform manual verification and link investigation later, thereby improving the observability of the system's operating status and the eventual consistency of the data.

[0121] See attached document Figure 6 As shown, in this embodiment, the baseline evolution and early warning module 40 is used to maintain early warning indicator standards in non-stationary environments and to handle monitoring blind spots and long-term memory management. This process can be divided into multiple execution logics.

[0122] The baseline evolution and early warning module 40 receives the final monitoring index value and the causal state token bound to it from the upstream operator. The baseline evolution and early warning module 40 parses the causal state token and extracts the baseline access permission bit carried within it. This baseline access permission bit indicates whether the current data belongs to a normal sample generated under stable system operating conditions.

[0123] When the baseline admission permission bit is deemed valid, the baseline evolution and early warning module 40 performs route filtering based on the service key value built into the data packet, routing the data to the matching active baseline node in the tree-structured baseline. Upon reaching the active baseline node, the baseline evolution and early warning module 40 updates the mean and variance parameters of that node using an incremental iteration method. Because the data volume in stream processing scenarios is extremely large, recalculating the mean and variance using the entire dataset would pose a serious risk of memory overflow.

[0124] As a preferred approach, the baseline evolution and early warning module 40 employs an incremental iterative model for updating. The incremental iterative formula for the baseline parameters is as follows:

[0125] ;

[0126] ;

[0127] ;

[0128] In the formula, Indicates the first The baseline mean after the next update. Indicates the first The baseline mean after the next update. This indicates the current final monitoring indicator value. This represents the total number of valid samples accumulated at the current baseline node. Indicates the first The sum of squared deviations after the next update Indicates the first The sum of squared deviations after the next update Indicates the first The updated baseline variance.

[0129] To avoid in the total number of samples When the denominator is equal to 1, an abnormal situation occurs where the denominator is 0. The baseline evolution and early warning module 40 determines... When it equals 1, directly... Assign the value to 0, and in When the variance is greater than 1, the above variance calculation logic is activated. Through this incremental calculation process, the baseline evolution and early warning module 40 can achieve long-term accurate evolution of baseline parameters without retaining historical detailed data.

[0130] When the baseline access permission bit carried by the causal state token is determined to be invalid, it indicates that the current data is in a transitional recovery state, or belongs to the isolation state corresponding to timeout degradation or missing token. For this transitional recovery state data, directly mixing it into the original active baseline nodes for updates will cause the normal baseline statistical characteristics to be contaminated.

[0131] Therefore, in this embodiment, the baseline evolution and early warning module 40 generates independent profile statistical features for these data in the transition recovery state and suspends these independent profile statistical features into a review period. During this review period, the baseline evolution and early warning module 40 collects the statistical features of this batch of data separately in an isolated memory area. After the review period ends, the baseline evolution and early warning module 40 determines whether to execute batch merging logic or split and derive new baseline nodes based on the calculated structural offset and combined with multi-dimensional features. The structural offset calculation model called by the baseline evolution and early warning module 40 is as follows:

[0132] ;

[0133] In the formula, Indicates structural offset. This represents the sample mean of the statistical characteristics of independent profiles during the review period. This represents the baseline mean of the original active baseline nodes. This represents the baseline standard deviation of the original active baseline nodes. This represents the zero-prevention minimum constant (e.g., 0.00001), which is used to ensure the validity of division operations and prevent algorithm crashes when the original baseline standard deviation approaches 0.

[0134] After obtaining the structural offset, the baseline evolution and early warning module 40 compares it with a preset structural offset threshold. This structural offset threshold is determined based on the confidence interval theory of the normal distribution, and is set to 3 in this embodiment. When the structural offset is less than or equal to the structural offset threshold, it indicates that the transition data has been restored to the original operating condition, and the baseline evolution and early warning module 40 merges the statistical parameters in the independent contour statistical features into the original active baseline nodes.

[0135] To avoid false migration judgments caused by short-term extreme noise from device sensors, as a preferred approach, when the structural offset exceeds a structural offset threshold, the baseline evolution and early warning module 40 further extracts the effective sample density within the review period. This effective sample density is defined as the ratio of the total number of samples actually received within the review time window to the theoretical total number of samples. Only when this effective sample density exceeds a preset density tolerance lower limit (e.g., 0.8) does the baseline evolution and early warning module 40 finally confirm that a substantial operational condition migration has occurred in the system. At this point, the baseline evolution and early warning module 40 uses the original active baseline node as its parent node, splits and derives a completely new baseline node, and sets this new baseline node as the new active baseline node, thus presenting a multi-branched tree-like baseline structure in physical structure.

[0136] As business models evolve, corresponding branch nodes will emerge in the tree-like baseline structure. To prevent unlimited splitting from causing system memory bloat, the baseline evolution and early warning module 40 introduces an asynchronous pruning and recycling mechanism. When an active baseline node does not receive data from newly routed distributions within a continuous time window, the baseline evolution and early warning module 40 changes the node's status from active baseline to archived historical baseline.

[0137] For nodes in the archived historical baseline state, the baseline evolution and early warning module 40 performs periodic scans using an asynchronous background thread. If the cumulative duration of its archiving state exceeds the preset maximum survival time, the baseline evolution and early warning module 40 forcibly releases the memory space it occupies. In this embodiment, the maximum survival time is not a static constant, but is dynamically calculated based on the ratio of the system's available off-heap memory capacity allocated to the node to the average number of bytes occupied by a single baseline node, to ensure that the node reclamation rate dynamically adapts to the physical limitations of the underlying hardware.

[0138] When the system is in a degraded state or the data is still under review, the data currently being calculated lacks corresponding steady-state baseline parameters, causing conventional anomaly detection mechanisms to fail and creating monitoring blind spots. To ensure that the system has basic anomaly interception capabilities during this period, the baseline evolution and early warning module 40 executes a fallback early warning rule.

[0139] The baseline evolution and early warning module 40 performs read-only tracing along the tree-like baseline structure towards the root, extracting the mean and variance parameters of the steady-state parent node closest to the current evolution path. The baseline evolution and early warning module 40 uses the parameters of this steady-state parent node as a temporary judgment criterion and compares them with the data currently in the downgrade or review period for detection.

[0140] Specifically, the baseline evolution and early warning module 40 constructs a temporary early warning boundary interval by expanding the extracted parent node mean upwards and downwards by a preset multiple (e.g., 3 times) of the parent node standard deviation. If the current data deviates from this temporary early warning boundary interval, the baseline evolution and early warning module 40 outputs a temporary early warning signal downstream. Through this fallback rule, the baseline evolution and early warning module 40 maintains continuous dynamic early warning output capability during the baseline reconstruction transition period.

[0141] The present invention also provides a computer device, including: a processor and a memory, the memory storing a computer program executable by the processor, the computer program performing the method described above when executed by the processor.

[0142] The present invention also provides a storage medium storing a computer program, which is executed by a processor to perform the method described above.

[0143] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0144] To help those skilled in the art to understand the present invention more intuitively, a specific application example is given below, based on the real-time transaction data monitoring scenario during a major e-commerce platform promotion, and the relevant experimental verification process and comparative effects are described in detail.

[0145] In this application scenario, the system needs to collect real-time statistics on key monitoring indicators such as "effective order amount per second" and "abnormal transaction frequency" for all business lines across the network, and trigger millisecond-level alerts when these indicators fluctuate. To verify the actual performance of this solution, an experiment was conducted using a stream computing cluster consisting of 20 computing nodes, with real traffic replays captured during peak business periods serving as the test data source. The control group adopted the currently mainstream traditional stream processing architecture based on event-time watermarking and full recalculation mechanisms.

[0146] See attached document Figure 7 During the 10th to 15th minute of the experiment, the test script artificially injected localized network congestion, causing severe lag in the message queue of a specific computing partition. (See attached...) Figure 7 As the curve trajectory shows, traditional stream processing architectures rely heavily on the monotonically increasing global watermark. The lag in data on one side causes the downstream time window to fail to close for a long time, and the end-to-end latency of the entire system shows a sharp upward trend, exceeding the business tolerance limit of 12,000 milliseconds, resulting in a monitoring blind spot of several minutes.

[0147] In contrast, the state acquisition and routing module 10 of this invention keenly detects the sharp changes in message queue lag and the first derivative of throughput, quickly determines that the system has entered a malicious blocking state, and issues control tokens downstream. The consensus control and state separation module 20, upon physical clock timeout, decisively triggers degradation logic, forcibly stripping the baseline access permission of incomplete data and completing window settlement. (Appendix) Figure 7The latency curve of this proposed solution exhibited only minor fluctuations during this phase, remaining consistently near the timeout tolerance threshold of 3000 milliseconds throughout. This comparative result confirms that this solution effectively avoids system deadlock under partial paralysis conditions, ensuring the continuity of the business early warning link.

[0148] See attached document Figure 8 This experiment primarily assesses the memory management capabilities of downstream operator nodes when processing large-scale delayed data. As the traffic replay speed increases, the manually set percentage of delayed data gradually rises from 1% to 15%. (See attached...) Figure 8 The bar chart height distribution shows that traditional architectures, when dealing with delayed data, require caching the original detailed data in memory for extended periods for subsequent recalculation. This leads to an exponential increase in the memory consumption of the underlying state of operator nodes, triggering an OutOfMemoryError (OOM) when delayed data reaches 12%. In this solution, delayed order details are no longer fully cached. The consensus control and state separation module 20 extracts the delayed data into core values ​​and performs incremental folding aggregation in the residual storage area based on epoch signatures. Regardless of the increase in delayed data volume, the underlying layer maintains only a single algebraic scalar. The residual synthesis and covering module 30 directly implements scalar-free merging in subsequent calculations. (Appendix) Figure 8 This clearly demonstrates that even with an extreme latency rate of 15%, the memory utilization rate of this solution remains consistently below the warning line of 40% of the total node capacity, effectively eliminating the memory disaster caused by long-tail data.

[0149] See attached document Figure 9 This section focuses on verifying the adaptive evolution capability of the baseline evolution and early warning module 40 when facing drastic fluctuations in the overall business environment. During the midnight rush of e-commerce promotions (the 30-minute node in the figure), the total order throughput of the entire network shows an exponential jump. This instantaneous extreme traffic surge causes the back pressure of the underlying stream processing system to rise sharply and then fall back in a controlled manner. The state acquisition and routing module 10 then accurately determines it to be in a transitional recovery state.

[0150] Because it is in a transitional recovery state, the data of the corresponding batch has been deprived of its regular baseline access permission. The baseline evolution and early warning module 40 intercepts it and suspends it in an isolated memory area for review. (See attached...) Figure 9 The dynamic distribution of scatter points and baselines reveals that when the peak of the buying frenzy arrives, the actual monitored values ​​instantly deviate from the original steady-state range. Traditional static thresholds or simple moving average monitoring are highly susceptible to generating a storm of false alarms at this moment. However, this solution, through statistical analysis of the independent contour features of the isolated background area, calculates that the structural offset of this batch of data far exceeds the preset structural offset threshold, and the effective sample density meets the tolerance lower limit. This calculation result allows the system to mathematically confirm that a substantial shift in the overall business operation (i.e., the activation of the promotional mode) has occurred, rather than being caused by occasional extreme noise interference.

[0151] Based on the above determination, the baseline evolution and early warning module 40 uses the original steady-state node as the parent node, smoothly splits, and derives entirely new high-load active baseline nodes. (See attached...) Figure 9 The intuitive demonstration is that the warning tolerance boundary jumps upwards in sync with the surge in business traffic, accurately capturing the sudden increase in promotional order data flow without triggering any false alarms. However, at the 45-minute mark, the business chain encountered a genuine black market order-brushing attack. The baseline evolution and warning module 40 immediately and accurately captured this attack based on the evolved new baseline parameters and issued a warning. This experiment intuitively demonstrates that the tree-based baseline architecture, when dealing with structural migrations in complex business scenarios, can achieve a seamless transition to eliminate false alarms while rapidly establishing a new steady state to prevent missed detections.

[0152] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for real-time monitoring and early warning of operational data based on streaming computing, characterized in that, include: Collect physical operation indicators of the stream processing system access point, and determine the current operating state of the stream processing system based on the physical operation indicators; Obtain preset control permissions, encapsulate the current running state and control permissions into a status token, and deliver the status token to the designated downstream node according to the computing topology of the stream processing system; When the time window of the downstream node reaches the event time boundary, the consistency of the received status token is checked, and the time window is closed when the consistency check passes. In physical memory, the data that arrives normally and the data that arrives late within the time window are stored separately, and the data that arrives normally and the data that arrives late are aggregated into independent data digests respectively. Extract the data summary, perform recalculation-free algebraic synthesis to generate the target index result, and perform overwrite update on the historical records in the downstream storage medium based on the target index result; Based on the access permissions carried in the status token, the target indicator result is routed to the tree baseline structure to perform baseline update or isolation review, and an early warning result is output based on the updated tree baseline structure.

2. The method for real-time monitoring and early warning of operational data based on streaming computing according to claim 1, characterized in that, Determining the current operating state of the stream processing system based on the aforementioned physical operating indicators includes: Obtain the physical operation metrics for the same period, including message queue lag, computation topology backpressure ratio, and data output throughput. Calculate the first derivative of the throughput rate at the data output terminal to obtain the rate of change of throughput performance; The message queue lag, the calculated topology backpressure ratio, and the throughput performance evolution rate are input into a preset state machine model, which outputs the current running state as a stable state, a benign burst state, a malignant blocking state, or a transitional recovery state.

3. The method for real-time monitoring and early warning of operational data based on streaming computing according to claim 1, characterized in that, The state token is composed of an epoch signature, a scope signature, runtime parameters, a window closure permission bit, and a baseline admission permission bit. Delivering status tokens to designated downstream nodes according to the computational topology of the stream processing system includes: Map the scope signature to the associated physical computing slot, and query the dependency mapping table to obtain the list of associated nodes; The status token is pushed to the physical memory of the downstream node based on the list of associated nodes.

4. The method for real-time monitoring and early warning of operational data based on streaming computing according to claim 3, characterized in that, When the downstream node's time window reaches the event time boundary, the consistency of the received state token is checked, and the time window is closed when the consistency check passes, including: When the time window reaches the event time boundary, the time window is controlled to switch to a waiting state; Extract all state tokens corresponding to each node in the current list of associated nodes, and perform a global logical AND operation on the window closure permission bit carried in the state token; When the result of the global logical AND operation is true, the consistency check is deemed to have passed, and the time window is controlled to change from the waiting state to the closed confirmation state.

5. The method for real-time monitoring and early warning of operational data based on streaming computing according to claim 4, characterized in that, The method further includes: A physical timer is started when the time window enters a waiting state. When the waiting delay of the physical timer reaches the preset timeout tolerance threshold and the result of the global logical AND operation is false, the time window is forcibly controlled to switch to the closed confirmation state. During the lifetime of the time window, the baseline access permission position of the data corresponding to the unreached state token is set to invalid, and a downgrade label is added to the data corresponding to the unreached state token.

6. The method for real-time monitoring and early warning of operational data based on streaming computing according to claim 3, characterized in that, In physical memory, normally arriving data and delayed arriving data within the time window are stored separately, and the normally arriving data and delayed arriving data are aggregated into independent data digests, including: Allocate independent first and second storage areas in physical memory; Data arriving before the time window closes is stored in the first storage area, and data aggregation is performed to generate a normal adjudication summary. Data that arrives late after the time window closes is stored in the second storage area. The core numerical fields of the late-arriving data are extracted and incremental folding updates are performed according to the epoch signature to generate a residual incremental digest.

7. The method for real-time monitoring and early warning of operational data based on streaming computing according to claim 6, characterized in that, The method further includes: The maximum lifetime parameter is calculated based on the available memory capacity of the node and the fixed byte size of a single residual incremental digest; Compare the actual residence time of the residual increment summary with the maximum survival time parameter; Release the physical memory space occupied by the residual increment digest whose actual residence time is greater than or equal to the maximum lifetime parameter, and write the data that exceeds the maximum lifetime parameter into the dead letter queue.

8. The method for real-time monitoring and early warning of operational data based on streaming computing according to claim 6, characterized in that, Extract the data summary, perform recalculation-free algebraic synthesis, and generate the target index result, including: Extract the normal adjudication summary and residual increment summary that have the same epoch signature; The discarded scalars triggered by the withdrawal operation are subtracted from the residual increment summary, and the positive increment scalars triggered by the append operation are added. The normal decision summary and the residual increment summary are then merged to generate the target index result.

9. The method for real-time monitoring and early warning of operational data based on streaming computing according to claim 6, characterized in that, The method further includes: A unique state fingerprint is generated by combining the current normal decision digest, epoch signature, and concurrent subtask number of the current processing node for hash calculation. Construct provisional state data containing the epoch signature, unique state fingerprint, normal decision digest, and monotonically increasing logical version number, and output it to the downstream storage medium; Based on the target metric results, perform an overwrite update on the historical records in the downstream storage medium, including: Using the primary key identifier, scope signature, and unique state fingerprint as ternary verification conditions, a history record in the downstream storage medium is overwritten. If the overwrite verification fails, the monotonically increasing logical version number of the newly generated record is compared with that of the historical record. If the monotonically increasing logical version number of the newly generated record is greater than that of the historical record, the newly generated record is overwritten.

10. The method for real-time monitoring and early warning of operational data based on streaming computing according to claim 1, characterized in that, Based on the access permissions carried in the status token, the target metric result is routed to the tree baseline structure for baseline update or isolation review, including: When the access permission is valid, the target index result is routed to the active baseline node in the tree baseline structure. The node mean and sum of squared deviations are incrementally iterated using the current baseline mean, the number of arriving samples, and the total number of cumulative samples of the active baseline node to complete the baseline update. When the access permission is invalid, the current data is suspended to an isolated memory area to collect contour statistical features; Obtain the sample mean of the suspended samples, calculate the difference between the sample mean and the original baseline mean, and divide the difference by the original baseline standard deviation with an added zero constant to obtain the structural offset. When the structural offset is greater than the preset structural offset threshold, a new baseline node is derived from the original active baseline node as the parent node, and the new baseline node is set as the current active branch.

Images