A real-time data governance method based on flow feature perception dynamic window and state compensation

Abstract

The application provides a real-time data management method based on flow feature perception dynamic window and state compensation, comprising: acquiring flow data; dynamically adjusting the window boundary of a data quality evaluation calculation window through an adaptive control algorithm according to a running state parameter; executing non-blocking quality evaluation in the window and generating a window pollution mark; redirecting the data of a polluted window to an isolated sandbox according to the mark, and continuing the flow of un-polluted data; constructing a reverse compensation flow injection flow pipeline for abnormal data that has flowed into the downstream, and performing a state inverse operation on key state according to the reverse compensation flow injection flow pipeline, so as to erase the state pollution caused by the abnormal data. The application can improve the real-time performance, accuracy and system stability of flow data management.

Description

Technical Field

[0001] This invention relates to the field of real-time big data processing technology, and more specifically, to a real-time data governance method based on flow feature-aware dynamic windows and state compensation. Background Technology

[0002] With the acceleration of digital transformation, the scale of data generated by enterprise business systems is growing exponentially. Streaming data processing technology, due to its low latency and high throughput, has become a core infrastructure for real-time analysis, real-time monitoring, and real-time decision-making. Current mainstream streaming frameworks, such as Apache Flink and Apache Spark Streaming, effectively support the real-time flow and aggregation analysis of massive amounts of data through technological innovations such as event-time semantics, watermark mechanisms, and incremental computation. Regarding data quality governance, existing technologies typically employ offline batch processing or fixed-window online detection methods. These methods use predefined rules to verify the integrity, consistency, and accuracy of data, discarding or storing detected abnormal data in a dead-letter queue for subsequent manual processing. Some solutions attempt to embed simple filtering operators in the streaming processing chain to achieve real-time interception of abnormal data through static threshold judgment. Furthermore, state management, as a core capability of stream computing, is generally addressed in existing frameworks which use keyed state storage to maintain intermediate aggregation results and checkpoint mechanisms for fault recovery. However, effective automatic correction methods are still lacking for aggregation state contamination caused by abnormal data.

[0003] Existing detection methods using fixed windows or static thresholds cannot adapt to fluctuations in streaming data rates and out-of-order characteristics, resulting in high detection latency and a high false alarm rate. Once abnormal data is discarded or isolated, the downstream aggregation state pollution it has triggered cannot be automatically eliminated, causing continuous distortion of statistical results. There is a lack of a closed-loop mechanism to reintegrate abnormal data into the main processing link after repair, resulting in severe loss of data value. The reverse compensation operation lacks idempotency guarantees, which can easily lead to secondary state errors in retry or fault recovery scenarios. Summary of the Invention

[0004] This invention provides a real-time data governance method based on flow feature-aware dynamic windows and state compensation, comprising: Acquire real-time streaming data from the streaming data processing pipeline; Based on the operating status parameters of the streaming data processing pipeline, the window boundary of the calculation window used for data quality assessment is dynamically adjusted through an adaptive control algorithm; Within the computation window, a non-blocking quality assessment is performed on the real-time streaming data, and a window contamination flag for the computation window is generated based on the assessment results. Based on the window contamination flag, real-time streaming data belonging to contaminated computing windows is redirected to an isolation sandbox, while real-time streaming data belonging to uncontaminated computing windows continues to flow along the backbone of the streaming data processing pipeline. For abnormal data that has flowed downstream and participated in aggregation calculations before the window contamination marker is generated, a reverse compensation stream is constructed and the reverse compensation stream is injected into the streaming data processing pipeline. The downstream aggregation operator performs state inverse operation on the underlying keying state according to the reverse compensation stream, so as to erase the state pollution caused by the abnormal data.

[0005] Further, the step of dynamically adjusting the window boundary of the calculation window used for data quality assessment through an adaptive control algorithm based on the operating state parameters of the streaming data processing pipeline includes: Using the throughput rate, out-of-order delay time and back pressure rate of the streaming data processing pipeline as input variables, a system load function is constructed using a proportional-integral-derivative control algorithm. Based on the value of the system load function, the step size and window size of the sliding time window are elastically scaled. When the value of the system load function exceeds the preset high water level safety threshold, the step size of the sliding time window is reduced logarithmically or downgraded to a counting window. When the value of the system load function falls below a preset low water level threshold and the duration exceeds a preset damping period, the window size of the sliding time window is smoothly increased.

[0006] Further, the step of performing a non-blocking quality assessment on the real-time streaming data within the computation window, and generating a window contamination flag for the computation window based on the assessment result, includes: Within the calculation window, a probabilistic data structure is used to perform dimensionality reduction statistics on the feature fields of the real-time streaming data in order to extract data quality features; Based on the comparison results between the data quality characteristics and the adaptive dynamic threshold, it is determined whether the calculation window is contaminated; If the comparison result indicates that the computation window is contaminated, a window contamination tag is generated to mark the computation window as a contaminated window.

[0007] Furthermore, within the calculation window, a probabilistic data structure is used to perform dimensionality reduction statistics on the feature fields of the real-time streaming data to extract data quality features, including: The cardinality of the primary key field within the calculation window is calculated using the HyperLogLog algorithm to obtain the percentage of unique cardinality, which is the ratio of the unique cardinality after deduplication to the total number of records. The T-Digest algorithm is used to dynamically cluster the numerical fields of the real-time streaming data to obtain the distribution centroids, and the quantile values ​​are estimated based on the distribution centroids. The step of determining whether the calculation window is contaminated based on the comparison result between the data quality characteristics and the adaptive dynamic threshold includes: When the decrease in the percentage of independent bases exceeds a first preset threshold, or the sudden increase in the quantile value exceeds a second preset threshold, the calculation window is determined to be contaminated.

[0008] Further, the step of redirecting real-time streaming data belonging to contaminated computing windows to an isolation sandbox based on the window contamination flag, and continuing to allow real-time streaming data belonging to uncontaminated computing windows to flow along the backbone of the streaming data processing pipeline, includes: Configure dynamic conditional branches in the transformation operators of the streaming data processing pipeline; Identify the window contamination marker of the computing window to which the real-time streaming data belongs; If the window contamination flag indicates that it is not contaminated, the real-time streaming data is output to the downstream operator through the backbone link; If the window contamination flag indicates that the data is contaminated, the real-time streaming data is tagged with an anomaly and redirected to a dead-letter queue for caching via a side output stream mechanism.

[0009] Furthermore, the step of constructing a reverse compensation stream for abnormal data that has already flowed downstream and participated in aggregation calculations before the window contamination marker is generated, and injecting the reverse compensation stream into the streaming data processing pipeline, includes: Extract details of abnormal data that have flowed downstream and participated in aggregation calculations; Based on the details of the abnormal data and the semantics of the data change log, a reversal event carrying a negative operation identifier is constructed for each abnormal data. The data structure of the reversal event is the same as the original data structure of the abnormal data, but the numerical signs are reversed. The constructed inverted event is used as a broadcast stream and injected into the streaming data processing pipeline according to the original path of the abnormal data.

[0010] Furthermore, the step of constructing a reversal event carrying a negative operation identifier for each piece of abnormal data based on the abnormal data details and the semantics of the data change log includes: Extract the key value, timestamp, numeric field, and metadata information of each abnormal data entry from the abnormal data details; Based on the original operation type of the abnormal data, a corresponding reverse operation type is generated, and the reverse operation type and the original operation type form a mutually inverse semantic pair. The data record for the reversal event is constructed using the key value, the timestamp, the reverse operation type, and the inverted numerical field. The data record of the reversal event is associated and bound with the operator topology path through which the abnormal data originally flowed, to generate a reversal event carrying routing information; Align and calibrate the timestamp of the inverted event with the processing time when the original abnormal data flows through the downstream operator to generate an inverted event carrying a time alignment identifier; The reverse events carrying the routing information and the time alignment identifier are batch aggregated into a reverse compensation stream.

[0011] Furthermore, the downstream aggregation operator performs a state inverse operation on the underlying keying state based on the reverse compensation stream to erase state pollution caused by the abnormal data, including: After receiving the reversal event in the reverse compensation stream, the downstream aggregation operator locates the corresponding keying state in the underlying key-value state repository based on the key value in the reversal event. The current aggregate value is read from the keying state, and the state inversion operation is performed according to the value in the inversion event to update the aggregate value in the keying state. The state inversion operation is to subtract the absolute value of the value in the inversion event from the current aggregate value. The aggregated value in the keyed state is updated to the result of performing the state inverse operation to complete the state rollback.

[0012] Furthermore, it also includes a late replay closed-loop step of cleaning and repairing the abnormal data cached in the isolation sandbox and then re-injecting it into the streaming data processing pipeline: Read the abnormal data and its corresponding window contamination markers and abnormality type identifiers from the isolation sandbox; Based on the anomaly type identifier, the corresponding cleaning and repair rules are matched from the pre-configured cleaning strategy library. The cleaning and repair rules include at least field correction rules, format conversion rules, and missing value filling rules. The abnormal data is cleaned and repaired according to the matched cleaning and repair rules to generate repaired data; Perform a quality re-inspection on the repaired data, and verify whether the repaired data meets the preset quality access conditions based on the results of the quality re-inspection. If the repaired data meets the quality admission criteria, then the repaired data is marked as late and treated as a normal positive event, and re-injected into the streaming data processing pipeline according to the operator topology path through which the abnormal data originally flowed. If the repaired data does not meet the quality admission criteria, the repaired data will be marked as unrepairable data and transferred to the permanent abnormal storage area. After the repaired data is re-injected into the streaming data processing pipeline, the compensation result of the repaired data participating in downstream aggregation calculations is monitored, and a closed-loop confirmation log is generated based on the compensation result.

[0013] Furthermore, it also includes the step of performing idempotent processing on the reverse compensation flow: In the downstream aggregation operator, a deduplication state cache is maintained for each keying state. The deduplication state cache is used to record the unique event identifier of the reversed event that has been processed. Read the current reverse event from the reverse compensation stream, and extract the event unique identifier of the current reverse event. The event unique identifier is generated by combining the data source identifier, primary key value, original event timestamp and operation type. Use the unique identifier of the event as the key to query the deduplication status cache; If the unique identifier of the event exists in the deduplication state cache, the current reversed event is determined to be a duplicate event, the current reversed event is discarded, and the state inversion operation is not performed. If the unique identifier of the event does not exist in the deduplication state cache, the state inversion operation is performed, and after the state inversion operation is completed, the unique identifier of the event is written into the deduplication state cache. A time-to-live (TTL) is set for the unique event identifier in the deduplication state cache. When the TTL of the unique event identifier exceeds the TTL, the unique event identifier is deleted from the deduplication state cache.

[0014] The embodiments of the present invention have at least the following beneficial effects: 1. This invention uses throughput rate, out-of-order delay time, and backpressure rate as input variables to the system load function, and employs a proportional-integral-derivative (PID) control algorithm to continuously adjust the step size and window size of the sliding time window. This ensures that the boundary of the computation window dynamically adapts to real-time changes in data flow. When a sudden surge in traffic causes the system load to exceed the high-water level safety threshold, this invention automatically and rapidly shrinks the sliding step size logarithmically, even downgrading it to a counting window. This forces the amount of data backlog within a single window to be limited to a controllable range, thereby preventing cluster memory overflow and operator backpressure collapse caused by data accumulation within the window from the source. After the traffic peak subsides, the window boundary smoothly expands under the protection of the damping period, avoiding frequent oscillations in window size. The entire adjustment process requires no manual intervention. The system can maintain a stable memory usage level and an orderly processing rhythm under extreme scenarios of drastic load fluctuations, improving the survivability of the streaming data processing cluster under peak impacts and the long-term robustness of the system.

[0015] 2. By constructing a reversal event carrying a negative operation identifier as a reverse compensation stream injection into the data processing pipeline, the downstream aggregation operator locates the keyed state based on the key value in the reversal event and performs the state inverse operation. The absolute value of the current aggregation value is subtracted from the value of the reversal event to complete the state rollback. At the same time, by maintaining a deduplication state cache for each keyed state and setting a time-to-live, the idempotency of the reverse compensation operation is guaranteed. This effectively solves the problem that abnormal data cannot be automatically eliminated after triggering downstream aggregation state pollution, avoids continuous distortion of statistical results, and prevents secondary errors in the state in retry or fault recovery scenarios.

[0016] 3. By redirecting abnormal data to an isolation sandbox and matching cleaning and repair rules such as field correction, format conversion, and missing value filling from a pre-configured cleaning strategy library based on the abnormality type identifier, the repaired data undergoes a quality re-inspection. Once the quality access conditions are met, a late arrival label is added, and the data is re-injected into the streaming data processing pipeline according to the original topology path. At the same time, the compensation results are monitored to generate a closed-loop confirmation log. This effectively solves the problem of severe data value loss caused by the inability to reintegrate abnormal data into the main processing link after it is discarded. It realizes a complete closed loop of data governance and ensures the full utilization of data assets. Attached Figure Description

[0017] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein: Figure 1 This is a flowchart illustrating a real-time data governance method based on flow feature-aware dynamic window and state compensation, provided in an embodiment of the present invention. Detailed Implementation

[0018] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make the invention more thorough and complete, and to fully convey the scope of the invention to those skilled in the art.

[0019] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present invention can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0020] It should be noted that the number of any elements in the accompanying drawings is for illustrative purposes only and not as a limitation, and any naming is for distinction only and has no limiting meaning.

[0021] The following is for reference. Figure 1 , Figure 1 This is a flowchart illustrating a real-time data governance method based on flow feature-aware dynamic windows and state compensation, provided in an embodiment of the present invention. Figure 1 As shown, a real-time data governance method based on flow feature-aware dynamic windows and state compensation includes: S1: Acquire real-time streaming data in the streaming data processing pipeline; This process acquires real-time streaming data from a streaming data processing pipeline. A streaming data processing pipeline is a data processing topology built on a distributed stream computing engine. It consists of multiple stream processing operators connected by directed edges, forming a directed acyclic graph. Data flows continuously between operators in the form of event streams. Real-time streaming data refers to data records continuously generated by the business system, with events as the basic unit. Each data record contains key-value pairs, timestamps, numeric fields, and metadata information. In this implementation, Apache Flink is used as the distributed stream computing engine. Real-time streaming data is read from a data source through Flink's DataStream API. The data source can be an Apache Kafka message queue, Apache Pulsar, or a file system, etc. Taking Kafka as an example, the FlinkKafkaConsumer is configured to subscribe to a specified Kafka topic, and the deserialization mode is set to deserialize Kafka messages into Java or Scala POJO objects, thereby acquiring real-time streaming data. Flink jobs run in a distributed manner on the cluster, consuming data from Kafka partitions in parallel by setting the parallelism. Once the acquired real-time streaming data enters the Flink data stream, it can be processed in subsequent steps.

[0022] S2: Based on the operating status parameters of the streaming data processing pipeline, the window boundary of the calculation window used for data quality assessment is dynamically adjusted through an adaptive control algorithm; Based on the operational status parameters of the streaming data processing pipeline, an adaptive control algorithm dynamically adjusts the window boundaries of the computation window used for data quality assessment. Operational status parameters are key indicators characterizing the current operational load of the streaming data processing pipeline, including throughput, out-of-order latency, and backpressure rate. Throughput refers to the number of data records successfully processed by the streaming data processing pipeline per unit time. Out-of-order latency is the difference between the event time and processing time of a data record, used to measure the degree of data out-of-order. Backpressure rate refers to the pressure exerted on the upstream when the downstream operator's processing capacity is insufficient, with a value ranging from 0 to 1. The computation window is the time interval used for batch quality assessment of real-time streaming data, and its boundaries include the window size and sliding step size. The adaptive control algorithm is an algorithm that automatically adjusts control parameters according to the system's operational status to maintain stable system operation.

[0023] In practice, runtime status parameters are collected through various methods. Throughput rate is obtained by setting a counter at the data source access point to count the number of data records flowing in per second. Out-of-order delay time is calculated by comparing the event timestamp carried in each data record with the current system processing time, and calculating the difference between the two. Backpressure rate is obtained through Flink's monitoring API, specifically by collecting the usage rate of operator output buffers through Flink's Metric system; when the buffer usage rate consistently exceeds a threshold, it indicates backpressure. Flink's backpressure monitoring involves the JobManager periodically triggering backpressure sampling in the TaskManager, and the sampling results are exposed through the Web UI and REST API. The system periodically calls Flink's REST API to obtain the backpressure status of each operator, quantifying the backpressure rate as a value between 0 and 1.

[0024] Using throughput rate, out-of-order delay time, and back pressure ratio as input variables, a proportional-integral-derivative (PID) control algorithm is used to construct the system load function. PID is a classic closed-loop control algorithm consisting of proportional, integral, and derivative components. It generates the control quantity by calculating the proportional, integral, and derivative values ​​of the error. In practical implementation, the system load function is defined as the target control quantity, and throughput rate, out-of-order delay time, and back pressure ratio are used as input variables. The PID algorithm generates a window adjustment amount by calculating the deviation between the current system load and the target load in real time.

[0025] The system load function is shown in the following formula: L(t)=alpha·R(t)+ beta·D(t)+gamma·B(t) Where L(t) represents the system load function value at time t, R(t) represents the throughput rate at time t, D(t) represents the out-of-order delay time at time t, B(t) represents the back pressure ratio at time t, and alpha, beta, and gamma are the weighting coefficients for throughput rate, out-of-order delay time, and back pressure ratio, respectively. In a typical implementation, alpha is set to 0.3, beta to 0.3, and gamma to 0.4, with the sum of all weighting coefficients being 1.

[0026] Based on the system load function value, the sliding time window's step size and window size are dynamically adjusted. The sliding time window is a time interval that slides forward with a fixed step size to define a data range, comprising two parameters: window size and sliding step size. Flexible scaling refers to the process of dynamically adjusting the window parameters according to the system load. In practice, the initial values ​​for the window size and sliding step size are set to 60 seconds and 10 seconds, respectively. When the system load function value changes, the window adjustment amount is calculated using a proportional-integral-derivative (PID) control algorithm, and the window size and sliding step size are adjusted synchronously.

[0027] When the system load function exceeds a preset high-water level safety threshold, the sliding time window step size is reduced logarithmically, or it is downgraded to a counting window. The high-water level safety threshold refers to the maximum allowable load of the system, typically set to 0.85 in a typical implementation. Logarithmic reduction means reducing the step size according to a base-2 logarithmic scale, i.e., reducing the step size to half, a quarter, an eighth, etc. of the original step size. A counting window is a method of defining the window boundary based solely on the number of data records, independent of time; for example, a window calculation is triggered every 1000 data records received. In a specific implementation, when the system load function exceeds 0.85, dynamic shrinkage and degradation are triggered, reducing the sliding step size logarithmically, from 10 seconds to 5 seconds, 2.5 seconds, 1.25 seconds, and finally 1 second. Under extremely high load conditions, it directly switches to counting window mode, with a counting threshold set to 500 records, to quickly isolate bursts of dirty traffic with extremely small time slices.

[0028] When the system load function value falls below a preset low-water threshold and remains below it for more than a preset damping period, the sliding time window size is smoothly increased. The low-water threshold is the minimum allowable load limit for the system, typically set to 0.3. The damping period is a settling-off time to prevent frequent window oscillations, typically set to 30 seconds. Smooth increase refers to gradually increasing the window parameters in a progressive manner, with each increase being 1.2 times the current value, until the initial window size of 60 seconds and sliding step size of 10 seconds are restored. In specific implementations, smooth increase is triggered when the system load function value remains below 0.3 for more than 30 seconds, gradually restoring the window size and step size to ensure a sufficient statistical base under low load conditions.

[0029] S3: Within the computation window, perform a non-blocking quality assessment on the real-time streaming data, and generate a window contamination flag for the computation window based on the assessment results; Within the computation window, a non-blocking quality assessment is performed on the real-time streaming data, and a window contamination marker is generated based on the assessment results. Non-blocking quality assessment refers to an assessment method that does not block the normal flow of data or pause data processing during the assessment process. Quality assessment refers to the process of checking and measuring the data quality of real-time streaming data, including cardinality anomaly detection and distribution extremum detection. A window contamination marker is a flag used to identify whether the computation window is contaminated by abnormal data.

[0030] In practice, quality assessment logic is embedded in Flink's window operators. When a computation window triggers a computation, the window operator performs a quality assessment on all data records within the window. The assessment process is asynchronous and does not block the normal output of the data stream. When a window is triggered, in addition to performing normal aggregation computations, the quality assessment task is executed in parallel. The results of the quality assessment are appended to the window metadata in the form of window contamination tags.

[0031] Within the computation window, a probabilistic data structure is used to perform dimensionality reduction statistics on the feature fields of the real-time streaming data to extract data quality features. A probabilistic data structure refers to a data structure that significantly reduces memory usage at the expense of some accuracy. Feature fields refer to data fields used for quality assessment, including primary key fields and numeric fields. Dimensionality reduction statistics refer to the process of compressing massive amounts of data into a compact statistical summary. Data quality features refer to statistical indicators extracted from the data to determine data quality.

[0032] In practice, two probabilistic data structure instances are maintained for each computation window. One instance is used for cardinality calculation of the primary key field, and the other instance is used for quantile estimation of the numeric field. Both data structures are stored in memory in the heap memory of the Flink TaskManager, maintained independently for each window, and released when the window is closed.

[0033] Based on the comparison results between the data quality characteristics and the adaptive dynamic threshold, it is determined whether the calculation window is contaminated. The adaptive dynamic threshold refers to a judgment boundary that is dynamically adjusted according to the distribution of historical data, rather than a fixed static threshold.

[0034] If the comparison result indicates that the computation window is contaminated, a window contamination flag is generated to mark the computation window as a contaminated window. The window contamination flag is represented by a Boolean value, which takes the value of contaminated or uncontaminated. In specific implementations, the window contamination flag is output as part of the window metadata along with the window computation result for downstream operators to identify.

[0035] S4: Use the HyperLogLog algorithm to calculate the cardinality of the primary key field within the calculation window to obtain the percentage of independent cardinality; The HyperLogLog algorithm is used to calculate the cardinality of the primary key field within the calculation window, yielding the unique cardinality ratio. The HyperLogLog algorithm is a probabilistic cardinality estimation algorithm used to estimate the number of unique elements in a large dataset. A primary key field is a key field in a data record used to uniquely identify a record, such as order number, user ID, or device ID. Cardinality refers to the number of distinct values ​​in a dataset. The unique cardinality ratio is the ratio of the deduplicated unique cardinality to the total number of records.

[0036] In practice, the implementation scheme of Apache Commons Math library or HyperLogLog in Redis is adopted. The core idea of ​​the HyperLogLog algorithm is to apply a hash function to each input element, mapping the element to a binary hash value, and then estimate the cardinality of the set based on the longest length of leading zeros in the hash value. In the specific implementation, the precision parameter p of HyperLogLog is set to 14, corresponding to 2 to the power of 14, or 16384 registers, with a memory overhead of approximately 16KB. Each register stores the length of the longest observed sequence of leading zeros.

[0037] For each data record within the window, extract the value of its primary key field and apply the MurmurHash3128-bit hash function to generate a 64-bit hash value. Use the first p bits of the hash value as a register index, and the remaining 64-p bits are used to calculate the number of leading zeros. Compare the calculated number of leading zeros with the existing values ​​in the corresponding registers, and store the larger one in the register. When the window is closed, perform a harmonic average of all register values ​​to obtain an estimate of the independent cardinality.

[0038] The formula for calculating the percentage of independent base populations is as follows:

[0039] in, Indicates the percentage of independent bases. This represents the independent cardinality estimated by the HyperLogLog algorithm. This represents the total number of records within the window. The normal range for the independent cardinality percentage is between 0 and 1. Under normal data distribution, the independent cardinality percentage should remain relatively stable. When a large influx of duplicate data occurs, the independent cardinality percentage will experience a sharp drop.

[0040] The T-Digest algorithm is used to dynamically cluster the numerical fields of the real-time streaming data to obtain distribution centroids, and quantile values ​​are estimated based on these centroids. The T-Digest algorithm is a probabilistic data structure for estimating quantiles in streaming data. It approximates the overall distribution by clustering data points into multiple centroids. The distribution centroid is the cluster center formed by clustering adjacent data points in the T-Digest algorithm. Each centroid contains two attributes: mean (i.e., centroid position) and weight (i.e., the number of data points represented by that centroid). A quantile value refers to the value that ranks at a specific percentage when the dataset is arranged in ascending order. For example, the P99 quantile indicates that 99% of the data is less than that value.

[0041] In practical implementation, open-source T-Digest implementations are used, such as Twitter's Algebird library or Redis's T-Digest module. The compression factor is a key parameter controlling the balance between accuracy and memory overhead in the T-Digest algorithm; a larger compression factor results in higher accuracy but also higher memory consumption. In a typical implementation, the compression factor is set to 100. For each data record within the window, the value of its numeric field is extracted, and the `add` method of T-Digest is called to insert the value into the data structure. Internally, T-Digest maintains a set of distribution centroids, each recording the mean and weight. When a new data point is added, the algorithm merges it into the nearest centroid or creates a new centroid based on its proximity to existing centroids. Whether to merge depends on the current weight of the centroid and the maximum weight limit controlled by the compression factor. When the window is closed, the `quantile` method of T-Digest is called to estimate the specified quantile value, such as the P99 quantile value.

[0042] Based on the comparison results between the data quality characteristics and the adaptive dynamic threshold, it is determined whether the calculation window is contaminated. The calculation window is contaminated when the decrease in the independent cardinality percentage exceeds a first preset threshold, or when the sudden increase in the quantile value exceeds a second preset threshold. The first preset threshold is the upper limit to which the decrease in the independent cardinality percentage can be tolerated, typically set to 30%. For example, if the historical average independent cardinality percentage is 0.8, and the current window's independent cardinality percentage drops below 0.56 (a decrease exceeding 30%), an anomaly determination is triggered. The second preset threshold is the upper limit to which the sudden increase in the quantile value can be tolerated, typically set to 50%. For example, if the historical average P99 quantile value is 100, and the current window's P99 quantile value suddenly increases to above 150 (a sudden increase exceeding 50%), an anomaly determination is triggered. When any of the above conditions are met, the current calculation window is determined to be a contaminated window. The adaptive dynamic threshold is calculated based on the moving average of the data quality characteristics of the historical window, specifically using an exponentially weighted moving average method with a weight factor alpha of 0.2, so that the threshold can be slowly adjusted as the data distribution changes.

[0043] S5: Based on the window contamination marker, the real-time streaming data belonging to the contaminated computing window is redirected to the isolation sandbox, and the real-time streaming data belonging to the uncontaminated computing window continues to flow along the backbone of the streaming data processing pipeline; Based on the window contamination flag, real-time streaming data belonging to contaminated computing windows is redirected to an isolation sandbox, while real-time streaming data belonging to uncontaminated computing windows continues to flow along the main link of the streaming data processing pipeline. The isolation sandbox is an independent storage area used to isolate and store abnormal data; abnormal data is cached in this area without affecting the processing of normal data streams. The main link refers to the flow path of normal data in the streaming data processing pipeline.

[0044] In practical implementation, dynamic conditional branches are configured in the transformation operators of the streaming data processing pipeline. A transformation operator is a computational unit in the streaming data processing pipeline that transforms and processes data records. Dynamic conditional branches refer to logical judgment structures that dynamically select different output paths based on the attributes of the data records.

[0045] A window contamination flag is used to identify the computation window to which the real-time streaming data belongs. Each data record is assigned a window ownership identifier when it enters the streaming data processing pipeline, which points to the computation window to which the data record belongs. After the quality assessment of the computation window is completed, the window contamination flag is written to the window metadata store. Data records can determine the contamination status of their respective windows by querying the window contamination flag corresponding to their window ownership identifier.

[0046] If the window contamination flag indicates that it is not contaminated, the real-time streaming data is output to the downstream operator via the backbone link. The backbone link output refers to the normal data flow path, where data records continue to participate in subsequent business aggregation and calculation.

[0047] If the window pollution flag indicates pollution, the real-time streaming data is tagged with an anomaly and redirected to a dead-letter queue for caching via the side output stream mechanism. An anomaly tag is a marker appended to a data record to identify it as abnormal data. The side output stream is a multi-output stream mechanism provided by Apache Flink, allowing an operator to generate multiple output streams simultaneously. The dead-letter queue is a special queue within the message queue used to store messages that cannot be processed normally.

[0048] In the implementation, the side output stream logic is implemented in Flink's `ProcessFunction`. First, an `OutputTag` object is defined as the identifier for the side output stream. In the `processElement` method of `ProcessFunction`, for each input data item, the routing target is determined by querying its window pollution flag. If the window pollution flag is unpolluted, the `out.collect` method is called to send the data to the main output stream. If the window pollution flag is polluted, the `ctx.output` method is called to send the data to the side output stream. The side output stream is connected to a Kafka dead-letter queue topic, and anomalous data is written to the dead-letter queue for caching. The anomalous data cached in the dead-letter queue includes the complete original data record, anomaly tag, window pollution flag, and failure timestamp.

[0049] S6: For abnormal data that has flowed downstream and participated in aggregation calculation before the window contamination mark is generated, construct a reverse compensation stream and inject the reverse compensation stream into the streaming data processing pipeline; For anomalous data that has already flowed downstream and participated in aggregation calculations before the window contamination markers are generated, a reverse compensation stream is constructed and injected into the streaming data processing pipeline. The reverse compensation stream is a special data stream composed of reverse events, used to offset and counteract the impact of anomalous data on the downstream aggregation results.

[0050] In practice, details of abnormal data that have flowed downstream and participated in aggregation calculations are extracted. When the quality monitoring center determines that a certain calculation window is a contaminated window, it extracts details of all data records within that window from the window metadata storage. By comparing the processing timestamps of the data records with the generation timestamps of the window contamination markers, data records that had flowed out of the window operator, entered downstream, and participated in aggregation calculations before the window contamination markers were generated are identified. These data records are the abnormal data that need to be compensated.

[0051] Based on the details of the abnormal data, and using data change log semantics, a reverse event carrying a negative operation identifier is constructed for each abnormal data entry. Data change log semantics refers to the standard semantics used in database and data stream processing to describe data change operations, typically including operation types such as insert, update, and delete. A reverse event is an event that is numerically opposite to the original event and semantically inverse. The data structure of the reverse event is the same as the original data structure of the abnormal data, but with the opposite numerical sign. For example, if the original event is an insert operation with an amount of 100, then the reverse event is a delete operation with an amount of -100.

[0052] The constructed inverted event is used as a broadcast stream and injected into the streaming data processing pipeline according to the original path of the abnormal data. A broadcast stream is a special data stream that distributes the same data to all parallel subtasks, ensuring that each downstream operator instance receives the compensation data.

[0053] In practice, the injection of the reverse compensation stream is achieved through Flink's BroadcastStream mechanism. The constructed reverse event is serialized into a Flink data stream, converted into a broadcast stream using the broadcast method, and then connected to the main data stream using the connect method, so that each downstream operator instance can receive the reverse event.

[0054] In some embodiments, the key value, timestamp, numeric fields, and metadata information for each abnormal data record are extracted from the abnormal data details. The key value is a key field value in the data record used for grouping and aggregation, such as user ID, order number, etc. The timestamp is the time the data record was generated. Numeric fields are numeric fields in the data record that participate in aggregation calculations, such as transaction amount, number of clicks, etc. Metadata information is additional descriptive information about the data record, such as data source, data format version, etc.

[0055] Based on the original operation type of the abnormal data, a corresponding reverse operation type is generated. The original operation type refers to the data operation semantics carried by the abnormal data when it flows into the stream processing system, including three types: insertion, update, and deletion. The reverse operation type and the original operation type form a mutually inverse semantic pair. Specifically, the reverse operation type of the insertion operation is the deletion operation, the reverse operation type of the update operation is the rollback update operation (i.e., overwriting the updated value with the value before the update), and the reverse operation type of the deletion operation is the insertion operation.

[0056] In practice, an operation type mapping table is used to convert the original operation type to its inverse. The mapping table defines three sets of inverse pairs: INSERT... DELETE, UPDATE_BEFORE UPDATE_AFTER, DELETE INSERT. Here, UPDATE_BEFORE represents the old value before the update, and UPDATE_AFTER represents the new value after the update.

[0057] The data record for the reversed event is constructed using the key value, the timestamp, the reverse operation type, and the inverted numerical field. Inversion refers to reversing the sign of the numerical field, turning positive numbers into negative numbers and negative numbers into positive numbers. For example, if the original event's numerical field is 100, the inverted value will be -100.

[0058] The data record of the reversal event is associated and bound with the operator topology path through which the abnormal data originally flowed, generating a reversal event carrying routing information. The operator topology path refers to an ordered list of all operators through which the abnormal data travels from the source operator to the current downstream operator. The routing information ensures that the reversal event can flow through each operator along the exact same path as the original event.

[0059] The timestamp of the inverted event is aligned with the processing time of the original abnormal data as it flows through downstream operators, generating an inverted event carrying a time alignment identifier. This time alignment calibration ensures that the inverted event's processing order is consistent with the original event in the time dimension, avoiding state compensation errors caused by time discrepancies.

[0060] In practice, timestamp alignment is achieved by setting the event time of the reversed event to the processing time of the original event. At the same time, the reversed event is marked as a special event in the watermark processing so that it is not affected by the late data discarding policy.

[0061] The inversion events carrying the routing information and the time alignment identifier are batch-aggregated into a reverse compensation stream. Batch aggregation refers to packaging multiple inversion events into a batch for transmission to reduce network overhead and serialization overhead. In a specific implementation, each batch contains 100 inversion events, and batches are identified by batch numbers.

[0062] In some embodiments, the downstream aggregation operator performs state inverse operations on the underlying keyed state according to the reverse compensation stream to erase state pollution caused by the anomalous data. An aggregation operator is a computational unit in a streaming data processing pipeline that performs grouped aggregation calculations, such as summation, counting, and averaging. The underlying keyed state refers to state data indexed by key values, continuously maintained during streaming computation, and stored in the state backend. State inverse operations refer to mathematically inverse operations of the original aggregation operation; for example, if the original operation is addition, the inverse operation is subtraction.

[0063] In implementation, after receiving a reversal event from the reverse compensation stream, the downstream aggregation operator locates the corresponding keyed state in the underlying key-value state repository based on the key value in the reversal event. The key-value state repository refers to the underlying storage system used for persistently storing keyed states. In this specific implementation, Apache Flink's EmbeddedRocksDBStateBackend is used as the key-value state repository. RocksDB is an embedded key-value store database that uses a log merge tree as its index structure. In Flink, keyed states are stored in RocksDB as column families. The key consists of a serialized byte string of Keygroup, Key, and Namespace, and the value consists of serialized bytes of the state. Each keyed state is mapped to a column family, similar to a table in a traditional database. In this specific implementation, the keyed states in RocksDB are accessed through Flink's KeyedState interface. In KeyedProcessFunction, the ValueState object is obtained through the getRuntimeContext().getState method, which corresponds to a specific key-value pair in RocksDB.

[0064] The current aggregate value is read from the keyed state. Based on the value in the inversion event, the state inversion operation is performed to update the aggregate value in the keyed state. The state inversion operation is to subtract the absolute value of the value in the inversion event from the current aggregate value. For example, if the original aggregation operation is summation, the current aggregate value is 1000, and the absolute value of the value in the inversion event is 100, then the result of the state inversion operation is 900.

[0065] The aggregated value in the keyed state is updated to the result of performing the inverse state operation to complete the state rollback. In specific implementation, the updated aggregated value is written back to RocksDB using the `update` method of `ValueState`. The `update` method serializes the new value into a byte string and writes it to the corresponding column family and key in RocksDB. After the state rollback is completed, the output of the downstream aggregation operator is restored to the state before the abnormal data was processed, achieving millisecond-level seamless hot rollback erasure of the downstream contaminated state.

[0066] In some embodiments, the method further includes cleaning and repairing the abnormal data cached in the isolation sandbox and then re-injecting it into the streaming data processing pipeline. This refers to the complete process of re-injecting the cleaned and repaired data into the streaming processing system, participating in subsequent calculations, and achieving eventual data consistency.

[0067] During implementation, the anomalous data, along with its corresponding window contamination flag and anomalous type identifier, is read from the isolation sandbox. The dead-letter queue within the isolation sandbox stores complete anomalous data records and their context information. The consumer pulls anomalous data from the dead-letter queue, simultaneously reading the associated window contamination flag and anomalous type identifier. The anomalous type identifier is used to distinguish different anomalous categories, such as cardinality anomalies, quantile anomalies, etc.

[0068] Based on the anomaly type identifier, a corresponding cleaning and repair rule is matched from a pre-configured cleaning strategy library. The cleaning strategy library is a pre-configured set of cleaning rules categorized by anomaly type. The cleaning and repair rules include at least field correction rules, format conversion rules, and missing value imputation rules. Field correction rules are used to correct obvious errors in field values, such as replacing values ​​outside a reasonable range with default values ​​or correcting them through interpolation. Format conversion rules are used to convert field values ​​from one format to another, such as converting a date string from YYYYMMDD format to YYYY-MM-DD format. Missing value imputation rules are used to handle cases where field values ​​are empty, such as filling missing fields with the mean, median, or a specific default value.

[0069] In practice, the cleansing strategy library is implemented through configuration files or a rule engine. The configuration file, in YAML or JSON format, defines the mapping between exception types and cleansing rules. The rule engine can be an open-source engine such as Drools, supporting dynamic loading and updating of cleansing rules.

[0070] The abnormal data is cleaned and repaired according to the matched cleaning and repair rules to generate repaired data. The cleaning and repair operation sequentially performs field correction, format conversion, and missing value imputation. After each operation is completed, the status of the data record is updated, and finally, the repaired data is generated.

[0071] A quality re-inspection is performed on the repaired data, and the results are used to verify whether the repaired data meets the preset quality admission criteria. Quality re-inspection refers to re-exercising the same quality assessment process as step S3 on the repaired data, including cardinality detection and quantile detection. Quality admission criteria refer to the quality standards that the repaired data must meet, specifically: the proportion of independent cardinality must be no less than 70% of the normal mean, and the quantile value must not exceed 150% of the normal mean.

[0072] If the repaired data meets the quality admission criteria, it is marked as late and treated as a normal positive event, then re-injected into the streaming data processing pipeline along the original operator topology path of the abnormal data. The lateness flag is a special marker attached to a data record to inform the streaming system that the record is late and requires special handling. In Flink, late data is handled by setting `allowedLateness` and the side output stream. When the repaired data is re-injected, the event timestamps of the original data are used to ensure its correct position in the time dimension.

[0073] If the repaired data does not meet the quality admission criteria, it will be marked as unrepairable and transferred to a permanent anomaly storage area. The permanent anomaly storage area is a location used to permanently store unrepairable anomaly data, such as a designated directory in HDFS or S3, for subsequent manual analysis and processing.

[0074] After the repaired data is re-injected into the streaming data processing pipeline, the compensation result of the repaired data participating in downstream aggregation calculations is monitored, and a closed-loop confirmation log is generated based on the compensation result. The closed-loop confirmation log records the re-injection time, flow path, participating aggregation calculations, and final compensation result of the repaired data. By comparing the compensated aggregation value with the theoretical expected value, the eventual consistency of the data is verified. The closed-loop confirmation log is written to the logging system for monitoring and auditing purposes.

[0075] Furthermore, the method also includes idempotent processing of the reverse compensation flow; Idempotency means that executing the same operation multiple times produces the same result as executing it once. In a distributed stream processing environment, due to network retransmission, fault recovery, and other reasons, the same inversion event may be delivered to downstream operators multiple times. Without idempotency handling, repeatedly executing the state inversion operation will cause the aggregate value to be deducted multiple times, resulting in new data inconsistencies.

[0076] In the downstream aggregation operator, a deduplication state cache is maintained for each keyed state. The deduplication state cache records the unique event identifier of each processed inverted event. In practice, the deduplication state cache is also stored in the RocksDB state backend, with the keyed state as its scope, and is stored alongside the aggregated value state. The data structure of the deduplication state cache uses Flink's MapState, where the key is the unique event identifier and the value is the processing timestamp.

[0077] The current reversal event is read from the reverse compensation stream, and its unique event identifier is extracted. This unique event identifier is generated by combining the data source identifier, primary key value, original event timestamp, and operation type. The data source identifier is a unique number representing the data source, such as a combination of a Kafka topic name and partition number. The combination is generated by concatenating these four fields using colons as separators to form a string-based unique event identifier, for example: kafka_topic_orders:user_12345:2026-07-03T10:30:00.000Z:DELETE.

[0078] Use the unique identifier of the event as the key to query the deduplication status cache.

[0079] If the unique identifier of the event exists in the deduplication state cache, the current inverted event is determined to be a duplicate event, the current inverted event is discarded, and the state inversion operation is not performed. Discarding means not passing the inverted event to subsequent processing logic, but ignoring it directly.

[0080] If the unique event identifier does not exist in the deduplication state cache, the state inversion operation is performed, and after the state inversion operation is completed, the unique event identifier is written to the deduplication state cache. The operation of writing to the deduplication state cache and the state inversion operation are performed in the same transaction context to ensure that both either succeed or fail simultaneously, avoiding state inconsistency.

[0081] A time-to-live (TTL) is set for the unique event identifiers in the deduplication state cache. When the TTL of an event identifier exceeds the TTL, it is deleted from the deduplication state cache. The TTL refers to the maximum retention time of the event identifier in the deduplication state cache. In a specific implementation, the TTL is set to 7 days. After the TTL is exceeded, the system automatically cleans up expired event identifiers to prevent the deduplication state cache from growing indefinitely. The cleanup operation is implemented through Flink state timestamps and timer mechanisms. A cleanup timer is registered when the state is written, and the deletion operation is performed when the timer is triggered.

[0082] The following example illustrates the implementation of the present invention using a specific real-time transaction data governance scenario.

[0083] Suppose an e-commerce platform uses Apache Flink to build a real-time order statistics pipeline, grouping and aggregating orders by user ID to calculate each user's daily transaction amount and number of transactions in real time. This pipeline processes approximately 50,000 order records per second, with a 60-second sliding window and a 10-second step. Under normal business conditions, each time window contains approximately 3 million order records, the cardinality of unique user IDs remains stable at around 0.6, and the P99 percentile of order amount remains stable at around 5000 yuan.

[0084] During a marketing campaign, a criminal gang used automated scripts to create a large number of fake orders in a short period of time. These fake orders shared the same user ID pattern and each order was for 9,999 yuan. Approximately 500,000 fake order records flooded in within 10 seconds. At this point, system load monitoring detected a sharp increase in backpressure from 0.1 to 0.9, a sudden drop in throughput from 50,000 orders per second to 15,000 orders per second, and the system load function value exceeding the high watermark threshold of 0.85. This triggered dynamic window contraction, reducing the sliding step size from 10 seconds to 1 second. Within this tiny 1-second window, quality assessment detected a sharp drop in the percentage of independent data points from 0.6 to 0.05, a decrease of 91.7%, exceeding the first preset threshold by 30%. Simultaneously, the P99 percentile of the amount field jumped from 5,000 yuan to 9,999 yuan, a sudden increase of approximately 100%, exceeding the second preset threshold by 50%. Based on these combined factors, the window was determined to be a contaminated window, and a window contamination marker was generated. The side output stream, based on the window pollution flag, marks subsequent orders in the same batch with an anomaly tag and redirects them to the Kafka dead-letter queue for isolation and caching. Orders from normal users continue to flow along the main link to downstream aggregation operators, ensuring that normal business operations are not affected.

[0085] Meanwhile, approximately 5,000 fake order data entries that had crossed the water level and participated in downstream aggregation calculations within the previous one-second window had accumulated an erroneous transaction total of approximately 49,995,000 yuan and 5,000 erroneous transaction counts in the aggregation operator. The quality monitoring logic extracted the details of these already transmitted fake orders, constructed a reversal event for each fake order, used the same user ID from the fake order as the key, generated a reverse operation type DELETE from the original operation type INSERT, and used the inverted -9,999 yuan as the reverse value field. The reversal event was injected into the streaming data processing pipeline according to the operator topology path the fake order originally flowed through. Upon receiving the reversal event, the downstream aggregation operator located the corresponding keying status in the RocksDB key-value state repository using the user ID as the key, subtracted 9,999 yuan from the current aggregation value, and deducted the accumulated amount and number of 5,000 fake orders one by one, seamlessly restoring the aggregation result to the accurate state before the fake orders arrived.

[0086] Subsequently, the isolated fraudulent order data was read from the dead-letter queue. Based on the anomaly type identifier, the user ID validity verification rule in the field correction rules was matched, correcting the abnormal fraudulent user IDs to the normal user ID format, and correcting the single transaction amount of 9999 yuan to the maximum allowed value of 1000 yuan for the marketing activity. The corrected order data underwent quality re-inspection, confirming that the independent base ratio had recovered to 0.58 and the P99 percentile of the amount had recovered to 5100 yuan. After meeting the quality access conditions, a late arrival flag was added, and the data was re-injected into the streaming data processing pipeline according to the original topology. This batch of re-inspected and qualified data participated in subsequent aggregation calculations as normal positive events, incrementally accumulating the corrected total transaction amount and number of transactions, achieving a final data consistency closed loop. The entire governance chain of anomaly detection, isolation, rollback compensation, and repair replay was completed within 200 milliseconds. Core business reports remained available throughout the entire process, without any downtime or data interruption.

[0087] The above description is merely an explanation of some preferred embodiments of the present invention and the technical principles employed. Those skilled in the art should understand that the scope of the invention as described in the embodiments of the present invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present invention.

Claims

1. A real-time data governance method based on flow feature-aware dynamic windows and state compensation, characterized in that, include: Acquire real-time streaming data from the streaming data processing pipeline; Based on the operating status parameters of the streaming data processing pipeline, the window boundary of the calculation window used for data quality assessment is dynamically adjusted through an adaptive control algorithm; Within the computation window, a non-blocking quality assessment is performed on the real-time streaming data, and a window contamination flag for the computation window is generated based on the assessment results. Based on the window contamination flag, real-time streaming data belonging to contaminated computing windows is redirected to an isolation sandbox, while real-time streaming data belonging to uncontaminated computing windows continues to flow along the backbone of the streaming data processing pipeline. For abnormal data that has flowed downstream and participated in aggregation calculations before the window contamination marker is generated, a reverse compensation stream is constructed and the reverse compensation stream is injected into the streaming data processing pipeline. The downstream aggregation operator performs state inverse operation on the underlying keying state according to the reverse compensation stream, so as to erase the state pollution caused by the abnormal data.

2. The method according to claim 1, characterized in that, The step of dynamically adjusting the window boundary of the calculation window used for data quality assessment based on the operating status parameters of the streaming data processing pipeline using an adaptive control algorithm includes: Using the throughput rate, out-of-order delay time and back pressure rate of the streaming data processing pipeline as input variables, a system load function is constructed using a proportional-integral-derivative control algorithm. Based on the value of the system load function, the step size and window size of the sliding time window are elastically scaled. When the value of the system load function exceeds the preset high water level safety threshold, the step size of the sliding time window is reduced logarithmically or downgraded to a counting window. When the value of the system load function falls below a preset low water level threshold and the duration exceeds a preset damping period, the window size of the sliding time window is smoothly increased.

3. The method according to claim 1, characterized in that, Within the computation window, a non-blocking quality assessment is performed on the real-time streaming data, and a window contamination marker for the computation window is generated based on the assessment results, including: Within the calculation window, a probabilistic data structure is used to perform dimensionality reduction statistics on the feature fields of the real-time streaming data in order to extract data quality features; Based on the comparison results between the data quality characteristics and the adaptive dynamic threshold, it is determined whether the calculation window is contaminated; If the comparison result indicates that the computation window is contaminated, a window contamination tag is generated to mark the computation window as a contaminated window.

4. The method according to claim 3, characterized in that, Within the calculation window, a probabilistic data structure is used to perform dimensionality reduction statistics on the feature fields of the real-time streaming data to extract data quality features, including: The cardinality of the primary key field within the calculation window is calculated using the HyperLogLog algorithm to obtain the percentage of unique cardinality, which is the ratio of the unique cardinality after deduplication to the total number of records. The T-Digest algorithm is used to dynamically cluster the numerical fields of the real-time streaming data to obtain the distribution centroids, and the quantile values ​​are estimated based on the distribution centroids. The step of determining whether the calculation window is contaminated based on the comparison result between the data quality characteristics and the adaptive dynamic threshold includes: When the decrease in the percentage of independent bases exceeds a first preset threshold, or the sudden increase in the quantile value exceeds a second preset threshold, the calculation window is determined to be contaminated.

5. The method according to claim 1, characterized in that, The step of redirecting real-time streaming data belonging to contaminated computing windows to an isolation sandbox based on the window contamination marker, and continuing to allow real-time streaming data belonging to uncontaminated computing windows to flow along the backbone of the streaming data processing pipeline, includes: Configure dynamic conditional branches in the transformation operators of the streaming data processing pipeline; Identify the window contamination marker of the computing window to which the real-time streaming data belongs; If the window contamination flag indicates that it is not contaminated, the real-time streaming data is output to the downstream operator through the backbone link; If the window contamination flag indicates that the data is contaminated, the real-time streaming data is tagged with an anomaly and redirected to a dead-letter queue for caching via a side output stream mechanism.

6. The method according to claim 1, characterized in that, The step of constructing a reverse compensation stream for abnormal data that has already flowed downstream and participated in aggregation calculations before the window contamination marker is generated, and injecting the reverse compensation stream into the streaming data processing pipeline, includes: Extract details of abnormal data that have flowed downstream and participated in aggregation calculations; Based on the details of the abnormal data and the semantics of the data change log, a reversal event carrying a negative operation identifier is constructed for each abnormal data. The data structure of the reversal event is the same as the original data structure of the abnormal data, but the numerical signs are reversed. The constructed inverted event is used as a broadcast stream and injected into the streaming data processing pipeline according to the original path of the abnormal data.

7. The method according to claim 6, characterized in that, Based on the details of the abnormal data and the semantics of the data change log, the step of constructing a reversal event carrying a negative operation identifier for each abnormal data entry includes: Extract the key value, timestamp, numeric field, and metadata information of each abnormal data entry from the abnormal data details; Based on the original operation type of the abnormal data, a corresponding reverse operation type is generated, and the reverse operation type and the original operation type form a mutually inverse semantic pair. The data record for the reversal event is constructed using the key value, the timestamp, the reverse operation type, and the inverted numerical field. The data record of the reversal event is associated and bound with the operator topology path through which the abnormal data originally flowed, to generate a reversal event carrying routing information; Align and calibrate the timestamp of the inverted event with the processing time when the original abnormal data flows through the downstream operator to generate an inverted event carrying a time alignment identifier; The reverse events carrying the routing information and the time alignment identifier are batch aggregated into a reverse compensation stream.

8. The method according to claim 1, characterized in that, The downstream aggregation operator performs state inverse operations on the underlying keying state based on the reverse compensation stream to erase state pollution caused by the abnormal data, including: After receiving the reversal event in the reverse compensation stream, the downstream aggregation operator locates the corresponding keying state in the underlying key-value state repository based on the key value in the reversal event. The current aggregate value is read from the keying state, and the state inversion operation is performed according to the value in the inversion event to update the aggregate value in the keying state. The state inversion operation is to subtract the absolute value of the value in the inversion event from the current aggregate value. The aggregated value in the keyed state is updated to the result of performing the state inverse operation to complete the state rollback.

9. The method according to claim 1, characterized in that, Also includes: Read the abnormal data and its corresponding window contamination markers and abnormality type identifiers from the isolation sandbox; Based on the anomaly type identifier, the corresponding cleaning and repair rules are matched from the pre-configured cleaning strategy library. The cleaning and repair rules include at least field correction rules, format conversion rules, and missing value filling rules. The abnormal data is cleaned and repaired according to the matched cleaning and repair rules to generate repaired data; Perform a quality re-inspection on the repaired data, and verify whether the repaired data meets the preset quality access conditions based on the results of the quality re-inspection. If the repaired data meets the quality admission criteria, then the repaired data is marked as late and treated as a normal positive event, and re-injected into the streaming data processing pipeline according to the operator topology path through which the abnormal data originally flowed. If the repaired data does not meet the quality admission criteria, the repaired data will be marked as unrepairable data and transferred to the permanent abnormal storage area. After the repaired data is re-injected into the streaming data processing pipeline, the compensation result of the repaired data participating in downstream aggregation calculations is monitored, and a closed-loop confirmation log is generated based on the compensation result.

10. The method according to claim 1, characterized in that, Also includes: In the downstream aggregation operator, a deduplication state cache is maintained for each keying state. The deduplication state cache is used to record the unique event identifier of the reversed event that has been processed. Read the current reverse event from the reverse compensation stream, and extract the event unique identifier of the current reverse event. The event unique identifier is generated by combining the data source identifier, primary key value, original event timestamp and operation type. Use the unique identifier of the event as the key to query the deduplication status cache; If the unique identifier of the event exists in the deduplication state cache, the current reversed event is determined to be a duplicate event, the current reversed event is discarded, and the state inversion operation is not performed. If the unique identifier of the event does not exist in the deduplication state cache, the state inversion operation is performed, and after the state inversion operation is completed, the unique identifier of the event is written into the deduplication state cache. A time-to-live (TTL) is set for the unique event identifier in the deduplication state cache. When the TTL of the unique event identifier exceeds the TTL, the unique event identifier is deleted from the deduplication state cache.

Images