Real-time streaming data cleaning and anomaly detection method

Abstract

The invention relates to a real-time streaming data cleaning and anomaly detection method. The method comprises the following steps: assigning a timestamp and a serial number to input streaming data, and writing the input streaming data into a temporary storage domain to form a preset duration playback fragment; performing consistency and time sequence legality check on the temporarily stored data, generating an abnormal evidence set, marking the abnormal evidence set as an abnormal candidate if a trigger condition is met, and recording the evidence; generating an anti-certification condition according to the evidence, opening an anti-certification window for the abnormal candidate, and verifying the anti-certification condition by using a subsequent associated fragment; if the anti-certification condition is met in the anti-certification window, the abnormal candidates are revoked and serve as submitted cleaning data to be migrated to a submission domain to be output; and if the condition is not met and overtime occurs, determining abnormity and outputting the abnormity to an abnormal channel, and keeping evidence and a verification result. The risk of misjudgment and miskilling caused by data disorder, jitter or instantaneous inconsistency is effectively reduced, repeated revocation and re-judgment of abnormal candidates in an anti-verification window are avoided, and the interpretability, traceability and engineering controllability of data cleaning and anomaly detection results are improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a method for real-time streaming data cleaning and anomaly detection. Background Technology

[0002] As can be seen from existing technical documents, current methods for real-time streaming data cleaning and anomaly detection still have significant shortcomings and limitations in practical engineering applications. Taking the alarm suppression method based on event time and waterline time disclosed in CN114664062A as an example, the core focus of this solution is to alleviate alarm out-of-order problems by combining event time and waterline time. Its technical focus is mainly on alarm suppression and release strategies, rather than fine-grained cleaning and anomaly semantic judgment of the original streaming data itself. Although this method can avoid time out-of-order problems caused by data transmission delays to a certain extent, it is essentially still a window-level alarm scheduling mechanism. It lacks the ability to judge the internal structural consistency, field dependencies, and irreversible conflicts across sequences of a single streaming data. It does not provide an effective processing path for anomalies such as missing fields, incomplete semantics, or structural damage. Once the anomaly exceeds the scope of time out-of-order, the system may still continue to propagate semantically unreliable data downstream, thereby amplifying risks in real-time statistics or risk control scenarios. Meanwhile, the scheme uses fixed suppression and release conditions and water level boundaries. Whether an anomaly is established depends entirely on the overall judgment at the end of the window. It lacks the ability to manage intermediate states and make revocable decisions for anomaly candidates. Once an alarm is suppressed or released, the process lacks a traceable chain of evidence for the anomaly, which is not conducive to subsequent rule optimization and audit analysis.

[0003] On the other hand, document CN120045848A proposes an adaptive dynamic data cleaning method for real-time streaming data. While it improves the flexibility and efficiency of the cleaning process to some extent through business demand-driven approaches, multi-level classification, cleaning nodes, and feedback sampling mechanisms, this approach exhibits a clear path-driven characteristic. That is, the anomaly assessment results are mainly used to select or adjust the cleaning path, rather than constructing a clear anomaly judgment logic around a single stream of data. This method heavily relies on classification nodes, cleaning nodes, and machine learning models to assess the degree of anomalies. Although it possesses adaptive capabilities on a macro level, in scenarios with extremely high real-time requirements, its model training, evaluation, and path planning processes suffer from computational complexity and long response chains, making it difficult to guarantee the determinism and interpretability of anomaly judgments. Furthermore, this approach does not distinguish between reversible and irreversible anomalies. Anomalies that have irreversibly conflicted with previously submitted data may still be included in the cleaning path for attempted repair, easily leading to invalid calculations or even introducing new data inconsistencies. Meanwhile, its feedback sampling mechanism focuses more on the dynamic adjustment of sampling rate and resource utilization, but fails to form a rebuttal window, minimum withdrawal condition or short-circuit verification mechanism around abnormal evidence. This results in the abnormal judgment process lacking clear logical boundaries. Once the abnormal state fluctuates repeatedly at different nodes, the system can hardly avoid repeated cleaning and repeated verification, thus bringing additional performance overhead and instability in high-throughput real-time streaming environments.

[0004] In summary, existing technologies generally suffer from problems such as insufficiently refined semantic characterization of anomalies, irrevocable or untraceable anomaly determinations, and difficulty in balancing real-time performance with computational complexity. In particular, there is a lack of a data cleaning and anomaly detection method that can perform evidence-based management of anomalies in streaming scenarios, support rebuttal and revocation, and balance certainty and efficiency. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time streaming data cleaning and anomaly detection method, thereby addressing some of the drawbacks and shortcomings pointed out in the background art.

[0006] The present invention adopts the following technical solution to solve the above-mentioned technical problems: a real-time streaming data cleaning and anomaly detection method, comprising: assigning timestamps and sequence numbers to the input streaming data and writing them into a temporary storage field to form a playback segment of a preset duration;

[0007] Perform consistency and time sequence legality checks on the temporary data, generate an abnormal evidence set, and mark the data as an abnormal candidate and record the evidence if the triggering conditions are met.

[0008] Based on the evidence, generate the counter-evidence conditions and open the counter-evidence window for the abnormal candidate. Use subsequent related fragments to verify the counter-evidence conditions. If the counter-evidence conditions are met within the counter-evidence window, cancel the abnormal candidate and migrate it to the submission domain as clean data that can be submitted. If the conditions are not met and timeout occurs, confirm the abnormality and output it to the abnormal channel, while retaining the evidence and verification results.

[0009] Furthermore, the abnormal evidence set is divided into rebuttable evidence and non-rebuttable evidence; non-rebuttable evidence includes at least evidence of structural damage that cannot be uniquely recovered from redundant fields within the same data, or evidence that forms an irreversible conflict with the data that has been migrated to the submission domain in terms of sequence identifier; when the abnormal evidence set contains non-rebuttable evidence, the abnormal candidate is directly confirmed as abnormal data and output to the abnormal channel, and the rebuttal window is not enabled.

[0010] Furthermore, the generation of the disproving conditions includes generating corresponding dissolution fact conditions for each evidence item in the abnormal evidence set, and forming at least one minimum revocation subset; performing short-circuit verification on the minimum revocation subset in a preset order within the disproving window; revoking the abnormal candidate when any minimum revocation subset is established, and migrating the corresponding data to the submission field output as submitable cleaned data.

[0011] Furthermore, after marking the abnormal candidates and recording the abnormal evidence set, evidence locking is performed to keep the abnormal evidence set and the disproving conditions unchanged during the disproving window; newly appearing evidence records within the disproving window are incremental evidence and are not incorporated into the abnormal evidence set.

[0012] Furthermore, the formation of the minimum revocation subset includes performing dependency constraint checks on the evidence items in the abnormal evidence set, eliminating redundant evidence items that are logically implied by other evidence items, and retaining only the dissolution fact conditions corresponding to the evidence items that are not mutually implied, so as to generate at least one minimum revocation subset without redundant conditions.

[0013] Furthermore, the preset order is arranged according to the irreversibility of the evidence items. First, the dissolution fact conditions that can uniquely dissolve structural evidence are verified, and then the dissolution fact conditions used to dissolve temporal evidence are verified. When a dissolution fact condition that has been verified within the disproving window contradicts the subsequently arriving fragment, the verified dissolution fact condition is revoked and short-circuit verification continues until any minimum revocation subset is established or the disproving window times out.

[0014] Furthermore, the subsequent related fragments obtained within the counter-evidence window are limited to fragments that have the same sequence identifier as the abnormal candidate or are in a preset sequence interval and involve the corresponding field of the abnormal evidence set; when any minimum revocation subset is established and the abnormal candidate is revoked, the data migrated to the submission field output carries a revocation basis identifier, and the revocation basis identifier includes at least the minimum revocation subset identifier that is satisfied and its corresponding resolution fact condition identifier.

[0015] Furthermore, in the preset sequence, a corresponding commitment lock interval is set for each disproven fact condition to be verified; taking the streaming data within the preset observation window W containing the verification start point of the disproven fact condition as the statistical object, the length L of the commitment lock interval is calculated by the following formula:

[0016]

[0017] in, The maximum backwards of the sequence number of any subsequently arriving segment within the observation window W relative to the current sequence number of the data to be verified; Defined as the range of the difference between the arrival time and the timestamp within the observation window W, that is... , , The time it takes for the fragment to arrive at the system. The timestamp carried by the fragment; The proportion of the number of segments identified as conflicting within the observation window W to the total number of segments in the observation window; To preset the time granularity, The preset coefficient is used; the commitment lock interval is limited by the length L to cover the sequence identifier range or timestamp range on which the dissolution fact condition depends; the dissolution fact condition is confirmed as verified only when no subsequent arriving segment that conflicts with the dissolution fact condition appears within the commitment lock interval; if a conflicting segment appears, the dissolution fact condition is determined to have failed and the short-circuit verification continues.

[0018] Furthermore, when the verified factual condition for resolving conflict contradicts the subsequently arriving fragment and is revoked, the subsequently arriving fragment is recorded as a conflict fragment, and a local masking rule for the preset order is generated based on the conflict fragment, so that within the same counter-evidence window or a subsequent preset number of counter-evidence windows, the factual condition for resolving conflict fragments associated with the evidence items is verified first.

[0019] Furthermore, when generating the local masking rule, a conflict fingerprint is calculated for the conflict fragment. The conflict fingerprint includes at least the sequence identifier range of the conflict fragment, the set of fields involved, and the corresponding evidence item identifier. When a conflict fragment matching the conflict fingerprint appears in the subsequent disproving window, the verification of the dissolution fact condition corresponding to the evidence item identifier is skipped, and the verification order is switched to prioritize the verification of the dissolution fact condition used to dissolve the evidence item associated with the conflict fingerprint.

[0020] The beneficial effects of this invention are as follows: The real-time streaming data cleaning and anomaly detection method proposed in this invention introduces a temporary review mechanism, an anomaly evidence set, and a rebuttal window into the judgment process of streaming data processing. This eliminates the reliance on one-time, irreversible, immediate judgments for anomaly identification, effectively reducing the risk of false judgments and false rejections caused by data disorder, jitter, or momentary inconsistencies. By distinguishing anomaly evidence into rebuttable and non-rebuttable evidence, and achieving efficient verification of rebuttal conditions based on the minimum revocation subset and short-circuit verification mechanism, unnecessary computational overhead is significantly reduced while ensuring real-time performance. Simultaneously, by leveraging evidence locking, commitment locking intervals, and conflict fragments and conflict fingerprint-driven local masking rules, the rebuttal process is made stable and convergent, avoiding repeated revocation and rejudgment of anomaly candidates within the rebuttal window. Furthermore, both the anomaly revocation and confirmation processes retain clear evidence and basis identifiers, improving the interpretability, traceability, and engineering controllability of the data cleaning and anomaly detection results. This method is particularly suitable for high-concurrency, time-series-dependent real-time streaming data processing scenarios. Attached Figure Description

[0021] Figure 1 This is a logic diagram of the real-time streaming data anomaly determination based on abnormal evidence and counter-evidence windows in this invention. Figure 2 This is a diagram showing the relationship between the classification of abnormal evidence and the function of rebuttal verification in this invention. Figure 3 This is a schematic diagram illustrating the state evolution of the elimination of factual conditions during the verification process of the present invention by contradiction; Figure 4 This is a schematic diagram illustrating the determination of irrefutable abnormal evidence and direct confirmation of abnormalities in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the time offset fallback process within the counter-evidence window in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram illustrating the construction of the minimum undo subset and subsequent screening of related fragments in Embodiment 1 of the present invention; Figure 7 This is a schematic diagram illustrating the verification sequence of the factual conditions for the elimination of evidence items based on the irreversibility of evidence items in Embodiment 2 of the present invention; Figure 8 This is a schematic diagram illustrating the calculation of the length of the committed locking interval and its contribution ratio in Embodiment 2 of the present invention. Figure 9 This is a schematic diagram of the temporal offset distribution of conflict segments and the generation of conflict fingerprints in Embodiment 2 of the present invention. Detailed Implementation

[0022] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] Combined with appendix Figure 1 This invention discloses a real-time streaming data cleaning and anomaly detection method. The system processes received input streaming data. Streaming data can come from continuous data sources such as message queues, log acquisition channels, sensor data streams, or network event streams. When data enters the processing flow, the system assigns a unique timestamp and sequence number to each piece of streaming data. The timestamp represents the time sequence in which the data is received or generated in the system, and the sequence number identifies the sequential relationship of data within the same data stream. Subsequently, the streaming data with timestamps and sequence numbers is written to a temporary storage area for cache management. The temporary storage area stores streaming data arriving within a preset time period, thus forming a review segment that can be used for retrospective analysis. By setting a preset time period, the system maintains real-time processing capabilities while enabling short-term review of received data, providing necessary contextual information support for subsequent anomaly detection.

[0024] After streaming data is written to the temporary storage area, the system performs consistency and temporal validity checks on the data content in the temporary storage area. The consistency check determines whether the fields within a single data entry satisfy a preset logical relationship, or whether it is semantically consistent with other related data within the same review segment. The temporal validity check determines whether the data exhibits reverse order, jumps, or unreasonable temporal relationships in the sequence number or timestamp dimension. During these checks, the system records any anomalies as evidence and aggregates multiple pieces of evidence generated from the same streaming data entry into an anomaly evidence set. When the anomaly evidence set meets preset anomaly triggering conditions, the system marks the corresponding streaming data as an anomaly candidate and associates the anomaly evidence set with the anomaly candidate data for storage, serving as a crucial basis for generating counter-evidence conditions and determining anomalies in subsequent processing.

[0025] Once streaming data is marked as anomaly candidates and a corresponding set of anomalous evidence is formed, the system generates disproven conditions based on the set of anomalous evidence to determine whether the anomaly candidate is valid. Disproven conditions describe the factual state that can dissolve the anomalous evidence. Their generation process includes determining, for each piece of evidence in the anomalous evidence set, subsequent data features or temporal relationships that can invalidate or reasonably interpret that evidence. After generating disproven conditions, the system opens a disprovenance window for the anomaly candidate. This window limits the time or sequence range for verifying the disproven conditions and establishes a correlation with the anomaly candidate.

[0026] During the disproving window, the system continuously retrieves subsequent data fragments associated with the anomaly candidate from the temporary storage area. These subsequent data fragments are related to the anomaly candidate at least in terms of time sequence, sequence identifier, or field content. The system uses these subsequent data fragments to verify the disproving conditions. When a subsequent data fragment is detected to meet the disproving conditions, the system determines that the anomalous evidence has been resolved, thereby revoking the anomaly candidate. The streaming data corresponding to the revoked anomaly candidate is then migrated from the temporary storage area to the submission area as cleaned data to be submitted and output, in order to participate in subsequent normal data processing or storage processes.

[0027] If the counter-evidence window continues for the preset time and no subsequent data segment meeting the counter-evidence conditions is detected, the system determines that the abnormal evidence has not been resolved, thus confirming the abnormal candidate as abnormal data, and outputting it to the abnormal channel for separate processing or recording. In cases where an anomaly is confirmed, the system retains the set of abnormal evidence corresponding to the abnormal candidate and the verification results of the counter-evidence conditions for subsequent anomaly analysis, tracing, or rule adjustment.

[0028] Combined with appendix Figure 2 After generating the set of anomalous evidence, the system further classifies the evidence types within it into rebuttable and non-rebuttable evidence. Rebuttable evidence refers to evidence whose anomalous state can be resolved or reasonably explained by the arrival of subsequent data, while non-rebuttable evidence refers to evidence that cannot be resolved under the current data context and with subsequent data supplementation. By classifying the anomalous evidence set in this way, the system can distinguish between transient and deterministic anomalies, thereby avoiding unnecessary rebuttal verification processes for obviously invalid data.

[0029] Unreproducible evidence falls into at least two categories. The first category is structural damage evidence. When structural errors such as missing fields, duplicate fields, or broken field relationships occur in streaming data, and these errors cannot be uniquely recovered from redundant or equivalent fields within the same streaming data stream, the system will classify the corresponding evidence as unreproducible. The second category is sequence conflict evidence. When the streaming data to be tested forms an irreversible sequence conflict with data that has already been migrated to the submission domain and completed its output, and this conflict cannot be resolved by supplementing or correcting subsequent data, the system will classify the corresponding evidence as unreproducible.

[0030] When the set of abnormal evidence contains any irrefutable evidence, the system will not open a rebuttal window for the corresponding abnormal candidate. Instead, it will directly confirm the abnormal candidate as abnormal data and output it to the abnormal channel for isolation or recording. At the same time, the system retains the set of abnormal evidence and the judgment result corresponding to the abnormal candidate for subsequent anomaly tracing, statistical analysis, or optimization of processing strategies.

[0031] Once anomaly candidates are marked and corresponding sets of anomalous evidence are formed, the system generates rebuttal conditions to resolve the anomalies. Specifically, the system analyzes the cause of the anomaly for each piece of evidence in the set and determines rebuttal factual conditions that invalidate or reasonably explain each piece of evidence. Rebuttal factual conditions describe the objective factual state that indicates the anomalous evidence is no longer valid as subsequent data arrives. After generating rebuttal factual conditions for each piece of evidence, the system combines these conditions based on the dependencies and logical constraints between the evidence pieces to form at least one minimal rebuttal subset. This minimal rebuttal subset represents the smallest combination of conditions that can resolve the entire set of anomalous evidence without including redundant conditions.

[0032] After the disproving window opens, the system performs verification operations on the dissolution fact conditions in the minimum revocation subset according to the preset verification order. The verification process adopts a short-circuit verification method, that is, when it is found during the verification process that all dissolution fact conditions in a certain minimum revocation subset have been satisfied by subsequent associated data fragments, the system immediately determines that the minimum revocation subset is valid and terminates further verification of other minimum revocation subsets. At this time, the system revoks the corresponding anomaly candidate, migrates the streaming data corresponding to the anomaly candidate from the temporary storage domain to the submission domain as committable cleaned data, and outputs it, thereby ensuring that the data can be promptly restored to the normal processing flow after the anomaly is resolved.

[0033] After marking candidate anomalies and recording the set of anomaly evidence, the system synchronously performs an evidence locking operation to ensure that the set of anomaly evidence and the generated counter-evidence conditions remain unchanged during the counter-evidence window. Evidence locking prevents frequent changes in anomaly judgment conditions due to newly emerging anomalies during the counter-evidence verification process, thus ensuring the consistency and controllability of the counter-evidence logic. For new anomalies that emerge during the counter-evidence window, the system records their corresponding evidence as incremental evidence and stores it separately from the original set of anomaly evidence. Incremental evidence is not merged into the current set of anomalies but is used for subsequent anomaly analysis or the next round of anomaly judgment processing, thereby preventing oscillations or instability in the counter-evidence process.

[0034] After generating the abnormal evidence set and determining the dissolution factual conditions corresponding to each evidence item, the system further performs dependency constraint checks on the evidence items in the abnormal evidence set to construct a minimal subset of revocable evidence for verification by contradiction. The dependency constraint check analyzes whether there is a logical implication relationship between different evidence items, that is, whether the validity of one evidence item necessarily leads to the validity of another evidence item. Based on predefined evidence logic rules, the system compares and analyzes each evidence item in the abnormal evidence set to identify redundant evidence items that are logically covered or contained within other evidence items.

[0035] After completing the dependency constraint check, the system removes redundant evidence items from the anomalous evidence set, retaining only logically independent evidence items. Subsequently, the system combines only the resolving factual conditions corresponding to the retained evidence items to generate at least one minimal rescission subset free of redundant conditions. This minimal rescission subset significantly reduces the number of resolving factual conditions that need to be verified while ensuring the complete resolution of the anomalous evidence set, thereby reducing the computational complexity of the proof-of-contrast verification process.

[0036] After the disproving window is opened, the system does not perform disproving verification on all subsequent streaming data in the temporary storage area. Instead, it limits and filters the subsequent related fragments used for disproving verification. Specifically, the system only acquires subsequent data fragments that have the same sequence identifier as the anomaly candidate, or data fragments that are within a preset sequence interval and involve the corresponding fields in the anomaly evidence set. This ensures that the data participating in the disproving verification are directly related to the anomaly candidate in terms of both temporal relationship and semantic content. By limiting and filtering subsequent related fragments, the system can effectively reduce the interference of irrelevant data on the disproving verification process, reducing processing overhead while ensuring verification accuracy.

[0037] When any minimum revocation subset is detected within the proof-of-contrast window and an anomaly candidate is revoked accordingly, the system migrates the corresponding streaming data of the anomaly candidate as clean data to the submission domain for output. Simultaneously with the output migration, the system attaches a revocation basis identifier to this data, recording the basis for the anomaly revocation decision. The revocation basis identifier includes at least the identification information of the satisfied minimum revocation subset, and the corresponding identification information of each dissolving factual condition within that minimum revocation subset. By including the revocation basis identifier in the output data, subsequent data processing or analysis can clearly identify the cause and process of the anomaly revocation.

[0038] Combined with appendix Figure 3 When performing rebuttal verification on the minimum revocable subset, the system first sets a preset verification order for the corresponding dissolution factual conditions based on the degree of irreversibility of each evidence item in the anomalous evidence set. Irreversibility characterizes whether evidence, once established, can be corrected or dissolved by subsequent data. Structural evidence typically has high irreversibility, while chronological evidence has relatively low irreversibility. Based on this irreversibility determination, the system prioritizes verifying the dissolution factual conditions that can uniquely dissolve structural evidence, and then verifies the dissolution factual conditions used to dissolve chronological evidence. Thus, in the rebuttal verification process, the type of evidence that has the greatest impact on the anomaly determination is prioritized.

[0039] During the disproving evidence window, once a disproving fact condition is verified by a subsequent associated fragment, the system marks it as a verified disproving fact condition and continues with subsequent verification operations. If, during the arrival of subsequent data, the system detects a new associated fragment that contradicts a verified disproving fact condition, the system cancels the verified disproving fact condition and re-enters the short-circuit verification process. Through this cancellation and continuation verification mechanism, the system can promptly correct disproving evidence failures caused by subsequent data changes. The disproving evidence verification process will continue until all disproving fact conditions in any minimal cancellation subset are effectively verified, thereby canceling the abnormal candidate, or until the disproving evidence window reaches a preset timeout condition and verification is terminated, thus completing the abnormal confirmation process.

[0040] When verifying the disproven fact conditions in a preset order, the system sets a corresponding commitment lock interval for each disproven fact condition to be verified. This commitment lock interval is used to limit the effective range within which the disproven fact condition must maintain consistency after verification. The commitment lock interval is used to constrain whether subsequent data segments will conflict with the verified disproven fact conditions, thereby avoiding frequent changes in the disproven proof results due to out-of-order delivery or jitter in a streaming environment.

[0041] Specifically, the system uses a preset observation window that includes the starting point for verifying the facts of this resolution. Using the streaming data within the observation window as the statistical object, the length of the committed locking interval is adaptively calculated based on the temporal characteristics, disorder level, and conflict status of the data within the observation window. The calculation method is as follows:

[0042]

[0043] in, Indicates observation window The maximum backwards of the sequence number of any subsequently arriving segment relative to the current sequence number of the data to be verified is used to reflect the degree of disorder in the sequence dimension of the data stream; Indicates observation window The range of the time of arrival and the timestamp is used to reflect the jitter in the time dimension, and it is defined as follows:

[0044]

[0045] in,

[0046]

[0047] Indicates the first The time it takes for a fragment to arrive at the system. This indicates the timestamp carried by the segment; Indicates observation window The proportion of the number of segments identified as conflicting within the observation window to the total number of segments within the observation window is used to reflect the conflict density in the current environment of evidence against contradiction. This is a preset time granularity used to normalize time jitter. , , These are preset coefficients used to adjust the weighting of different factors on the length of the commitment lock-in interval.

[0048] The system is based on length Determine the coverage of the commitment lock interval. When the resolution of fact conditions relies on sequence identifiers for verification, the commitment lock interval coverage starts from the current data to be verified. A sequence identifier range; when the resolution of factual conditions relies on timestamp verification, the commitment to lock the interval coverage is based on the current verification start point. The timestamp range corresponding to the multiple time granularity. The system only confirms that the resolving fact condition has been verified if no subsequent arriving fragment conflicting with the resolving fact condition is detected within the commitment lock interval; if a conflicting fragment occurs within the commitment lock interval, the system determines that the resolving fact condition has failed and continues to execute the short-circuit verification process.

[0049] Once a disproving fact condition has been verified within the disproving evidence window, the system continuously monitors whether subsequent arriving related data fragments are consistent with that disproving fact condition. If the system detects a contradiction between a subsequent arriving fragment and the verified disproving fact condition in terms of field values, temporal relationships, or logical constraints, it determines that the disproving fact condition is invalid and immediately revokes its verified status. Simultaneously, the system records the subsequent arriving fragment that triggered this revocation as a conflicting fragment and establishes an association with the corresponding anomaly candidate and evidence item.

[0050] After recording conflicting fragments, the system generates local masking rules based on these fragments for a preset verification order. These local masking rules adjust the verification priority of disproving factual conditions within the current disproving window or a preset number of subsequent disproving windows, ensuring the system prioritizes verifying factual conditions that can resolve the evidence items associated with the conflicting fragments. By introducing local masking rules, the system avoids repeatedly verifying disproving factual conditions already negated by the conflicting fragments during subsequent disproving processes, thereby reducing invalid verifications and repeated reversals, and improving the convergence and overall processing efficiency of the disproving verification process.

[0051] When generating local masking rules based on conflict fragments, the system performs conflict fingerprint calculation on the conflict fragments to abstractly represent their key features. A conflict fingerprint includes at least three components: the sequence identifier interval corresponding to the conflict fragment, the set of fields involved in the conflict fragment, and the evidence item identifier associated with the conflict fragment. The sequence identifier interval describes the location range of the conflict fragment in the data stream, the set of involved fields characterizes the specific data fields that triggered the conflict, and the evidence item identifier indicates the type of anomalous evidence corresponding to the conflict fragment. The conflict fingerprint generated in this way can stably identify the features of conflict fragments without relying on the complete data content.

[0052] During the subsequent opening of the disproving window, the system continuously calculates conflict fingerprints for newly arriving data fragments and matches them with the already generated conflict fingerprints. When the system detects a conflict fragment matching a conflict fingerprint in a subsequent disproving window, it determines that there is a duplicate or similar conflict situation in the current disproving environment. At this point, the system no longer verifies the dissolving fact condition corresponding to the evidence item identifier, but directly skips the verification step of that dissolving fact condition and dynamically adjusts the verification order so that the disproving process prioritizes verifying other dissolving fact conditions that can dissolve the evidence item associated with the conflict fingerprint. By introducing a verification order switching mechanism based on conflict fingerprints, the system can effectively avoid repeatedly executing invalid verifications in the same conflict scenario, thereby improving the execution efficiency of the disproving process and enhancing the stability and consistency of the anomaly judgment results in the continuous disproving window.

[0053] Example 1:

[0054] In this embodiment, a real-time transaction log processing system continuously receives transaction event streams from multiple business nodes for subsequent risk control analysis and business statistics. Figure 4 As shown, streaming data uses a single transaction event as the basic processing unit. Transaction events arrive continuously along the sequence number direction. During this time period, the system received a total of 8 transaction events with sequence numbers ranging from 105228 to 105235, and the data arrival rate was approximately 10,000 transactions per second.

[0055] During system operation, at a certain moment, a transaction event data with sequence number 105230 was received in the temporary storage area, such as... Figure 4As indicated by the anomaly marker, the transaction event's fields show a completed transaction status, but the corresponding transaction amount field is missing, the spare amount field is empty, and the event timestamp field is present. During a consistency check of this transaction event, the system discovered that the structural relationship between the transaction status and transaction amount fields was disrupted. Since there are no redundant fields in this transaction event that can uniquely recover the transaction amount, the system records this anomaly as structural disruption evidence and adds it to the anomaly evidence set.

[0056] At the same time, the system further performs a timing validity check on the transaction event. Combined with... Figure 4 The sequence distribution of the submitted data shows that the sequence number of this transaction event, 105230, is smaller than the largest sequence number, 105235, of the transaction event that was previously migrated to the submission domain and successfully output. Since the data in the submission domain has already been used for downstream statistical analysis, the transaction event corresponding to sequence number 105230, even if it arrives later, cannot be corrected by adjusting the order, thus creating an irreversible conflict with the submitted data in terms of sequence identification. The system records this situation as sequence conflict evidence and adds it to the set of anomalous evidence.

[0057] After constructing the abnormal evidence set, the system classifies the evidence types within the set into rebuttable and non-rebuttable evidence. For the aforementioned structural damage evidence, due to the missing transaction amount field and the lack of a unique recovery path, the system determines this evidence to be non-rebuttable. Similarly, for the aforementioned sequence conflict evidence, because it creates an irreversible sequence conflict with data already migrated to the submission domain, the system also determines this evidence to be non-rebuttable. At this point, the abnormal evidence set contains two pieces of non-rebuttable evidence.

[0058] Based on the above judgment results, and in combination with Figure 4 The statistics below confirm that only sequence number 105230 was identified as an anomalous event among the current 8 transaction events. The system will no longer open a counter-evidence window for this transaction event, nor will it attempt to resolve the anomaly through subsequent streaming data. Instead, it will directly confirm the anomaly candidate corresponding to this transaction event as anomalous data and output it to the anomaly channel for isolation processing. Simultaneously, the system retains the set of anomalous evidence and the evidence type determination result corresponding to this anomaly candidate for subsequent anomaly statistics and rule optimization.

[0059] The system's temporary storage area subsequently received another transaction event data with sequence number 105242, such as... Figure 5As shown. The transaction event has the transaction number T202405180031, the user ID is U77891, the transaction status field is marked as completed, the transaction amount field is 1000, the reserve amount field is 1000, and the event timestamp is 1684387205123. After performing consistency and timing legality checks on this transaction event, the system found two pieces of anomalous evidence.

[0060] The first piece of anomalous evidence is state consistency evidence. The system detected that the transaction status of this event was completed, but in the previous transaction event with sequence number 105241 for the same user ID U77891, that user's transaction status was still in the processing state, and no intermediate completion confirmation event occurred. The second piece of anomalous evidence is time sequence offset evidence. For example... Figure 5 As shown in the broken line, the system detected that the arrival time of the transaction event was 1684387205140, which has an offset of 17 milliseconds from the event timestamp 1684387205123 it carries. This is significantly higher than the normal jitter threshold of 5 milliseconds observed by the system in this time period.

[0061] The system adds the two pieces of anomalous evidence to the anomalous evidence set and marks the transaction event as an anomalous candidate. Since neither of the anomalous evidence belongs to the structural damage category, and the sequence number of the transaction event, 105242, is greater than the maximum sequence number, 105235, which has been migrated to the submission domain, no irreversible sequence conflict is formed. Therefore, the system determines that the anomalous evidence set only contains rebuttable evidence, thus opening a rebuttal window for the anomalous candidate.

[0062] After the disproving evidence window opens, the system generates corresponding dissolution fact conditions for each piece of evidence based on the set of abnormal evidence. For state consistency evidence, the system generates the dissolution fact condition that a completion confirmation supplementary event with the same transaction number T202405180031 is received within the disproving evidence window, and the sequence number of this supplementary event is greater than 105242. For time-series offset evidence, the system generates the dissolution fact condition that a subsequent transaction event with the same user identifier U77891 is received within the disproving evidence window, and the offset between its arrival time and the event timestamp falls back to within 5 milliseconds.

[0063] After generating each resolution fact condition, the system performs dependency analysis on the evidence items in the abnormal evidence set, confirming that there is no logical implication relationship between the state consistency evidence and the time-series offset evidence. Therefore, the two corresponding resolution fact conditions together constitute a minimal revocation subset.

[0064] The system then performs short-circuit verification on the smallest revocation subset in a preset order within the disproving window. Since state consistency evidence is a structural semantic anomaly, its irreversibility is higher than that of temporal offset evidence; therefore, the system prioritizes verifying the dissolution factual conditions corresponding to state consistency. Within 120 milliseconds after the disproving window opens, the system receives a transaction supplement event with sequence number 105244, whose transaction number is also T202405180031, and the event type is completion confirmation. The system determines that this supplement event satisfies the dissolution factual conditions corresponding to state consistency and marks the condition as verified.

[0065] Subsequently, the system continued to verify the resolution factual conditions corresponding to the time series offset. Combined with... Figure 5 Subsequent data points show that during the duration of the counter-evidence window, the system received six transaction events with the same user identifier U77891 within the sequence number range of 105245 to 105250. The maximum offset between the arrival time and the event timestamp was 3 milliseconds, which is less than the preset threshold of 5 milliseconds. Based on this, the system determined that the factual condition for resolving the time-series offset evidence was met, thus confirming that the time-series offset evidence could be revoked, and marked the corresponding factual condition for resolving the time-series offset evidence as verified.

[0066] Once both of the aforementioned conditions for resolving the factual issues are verified, the system determines that the minimum revocation subset is valid and immediately revokes the abnormal candidate with sequence number 105242. Subsequently, the system migrates this transaction event from the temporary storage domain to the submission domain as committable cleaned data and outputs it for subsequent transaction statistics and analysis.

[0067] During the entire disproving evidence window, the system performs an evidence locking operation on the abnormal evidence set corresponding to the abnormal candidate, keeping the abnormal evidence set and the disproving evidence conditions generated based on it unchanged. During this period, the system detects a network latency alarm event in the temporary storage domain that is not directly related to the transaction event. This event is recorded by the system as incremental evidence, but it is not incorporated into the current abnormal evidence set, nor does it affect the current disproving evidence verification process.

[0068] After completing the disproven verification process for transaction event with sequence number 105242 in the temporary storage area, the system enters the stage of constructing the minimum revocation subset and summarizing the verification results, such as... Figure 6 As shown. For this anomaly candidate, the system has formed an anomaly evidence set, which includes two items: state consistency evidence and time series offset evidence. After generating the corresponding resolving fact conditions, the system performs dependency constraint checks on the evidence items in the anomaly evidence set to determine whether there is a logical implication relationship between the evidence items.

[0069] Specifically, system analysis revealed that the validity of state consistency evidence does not necessarily lead to the validity of time-series shift evidence, and vice versa; the two are logically independent. Furthermore, the system determined that no third piece of evidence could simultaneously cover the anomalous semantics of the two aforementioned pieces of evidence. Therefore, the system did not find any redundant evidence items logically implied by other evidence items, nor did it detect any inclusion or substitution relationships between the dissolving factual conditions. Based on this dependency constraint check result, the system retains only the two dissolving factual conditions corresponding to the state consistency evidence and the time-series shift evidence, and combines them to generate a minimal revocation subset without redundant conditions. This minimal revocation subset is assigned a unique identifier, MS001, for subsequent rebuttal verification and revocation recording.

[0070] During the duration of the disproving window, the system does not verify all subsequent transaction events arriving in the temporary storage area, but rather limits and filters subsequent related fragments. For example... Figure 6 As shown in the bar chart, the system only acquires data fragments with the same transaction number T202405180031 as the anomaly candidate, or data fragments with sequence numbers between 105242 and 105260 that involve transaction status or timestamp fields. Through this constraint, the system filters out 9 subsequent related fragments within the disproving window. Of these, 7 are used to verify the state consistency disproving fact condition, and 2 are used to verify the time-series offset disproving fact condition. The number of fragments only indicates the number of related fragments participating in the verification, not the minimum number of fragments required for the disproving fact condition to be met. Other irrelevant transaction events are not included in the disproving verification scope, thus significantly reducing the computational load of the disproving verification.

[0071] After completing the proof by contradiction, the system confirms that both resolution fact conditions in the minimum revocation subset MS001 are met, and thus revokes the abnormal candidate with sequence number 105242. Subsequently, the system migrates this transaction event from the temporary storage domain to the commit domain as committable clean data and outputs it. While performing the migration output, the system attaches a revocation basis identifier to the transaction event to record the specific basis for the abnormal revocation. The revocation basis identifier includes at least the minimum revocation subset identifier MS001, and the corresponding resolution fact condition identifiers SC01 and SC02, where SC01 represents the state consistency resolution fact condition, and SC02 represents the time offset resolution fact condition.

[0072] In the submission domain, when the downstream system receives the transaction event, it can use the revocation basis identifier to clarify that the data was marked as an abnormal candidate and revoked by counter-evidence. This allows the system to audit or trace the data when needed, further verifying the interpretability and engineering applicability of the real-time streaming data cleaning and anomaly detection method of the present invention in actual business scenarios.

[0073] Example 2:

[0074] Based on Example 1, the temporary storage domain continuously receives transaction event streams from multiple business nodes. Each transaction event includes fields such as transaction number, user identifier, transaction status, transaction amount, event timestamp, and event sequence number. Figure 7 As shown, transaction events enter the system sequentially along the sequence number direction, which is used to illustrate the verification order and status changes of different dissolution fact conditions in the proof-of-contrast window.

[0075] After performing consistency and timing validity checks on transaction event 105318, the system detected three pieces of anomalous evidence: one structural anomaly and two temporal anomalies. The structural anomaly is that the transaction status field is marked as completed, but the corresponding settlement identifier field is missing, and there is no redundant field in the transaction event that can be used to uniquely deduce the settlement identifier. The two temporal anomalies are that the event timestamp has a 19-millisecond offset relative to the arrival time, and that there is an abnormal sequence interval between this transaction event and the previous transaction event under the same user identifier. The system added these anomalous pieces of evidence to the anomalous evidence set and marked the transaction event as an anomaly candidate.

[0076] After generating disproving conditions and opening the disproving window for the anomaly candidate, the system arranges the verification order of each disproving factual condition according to the degree of irreversibility of the evidence items in the anomaly evidence set. The system determines that structural anomaly evidence, once it cannot be disproven, will lead to the irreversibility of the transaction semantics, and its irreversibility is higher than that of temporal anomaly evidence. Therefore, if... Figure 7 As shown in the topmost bar, the system first verifies the dissolution fact condition used to uniquely resolve structural anomaly evidence in a preset order. This dissolution fact condition is defined as receiving a supplementary settlement event corresponding to the same transaction number within the disproving evidence window, and the supplementary event carrying a complete settlement identifier and passing the consistency check.

[0077] Within 90 milliseconds of the opening of the counter-evidence window, the system received a supplementary transaction event with sequence number 105320. Its transaction number matched the anomaly candidate and carried the complete settlement identifier field. Combined with... Figure 7 The system determines that the supplementary event meets the dissolution fact condition for dissolving structural anomalous evidence based on the continued validity interval corresponding to the structural dissolution fact condition, and marks the dissolution fact condition as verified and not revoked in the subsequent rebuttal process.

[0078] After verifying the structurally resolved factual conditions, the system continues to perform short-circuit verification of the resolved factual conditions corresponding to the two temporally anomalous pieces of evidence in a preset order. For example... Figure 7As shown in the middle and bottom bars, the first condition for time-series resolution is defined as receiving subsequent transaction events with the same user identifier within the disproving window, with the offset between their arrival time and the event timestamp falling back to within 5 milliseconds. The second condition for time-series resolution is defined as the interval between subsequent transaction events received within the disproving window returning to the system's preset normal range.

[0079] During the period of the disproving window, the system first verifies the first condition for the temporal resolution of facts. Within the range of sequence numbers 105321 to 105326, the system received a total of 6 transaction events with the same user identifier, with a maximum time offset of 4 milliseconds, which is less than the 5 millisecond threshold. This satisfies the first condition for the temporal resolution of facts, and the system marks this condition as verified.

[0080] However, in the subsequent transaction event with sequence number 105327, such as Figure 7 As shown, the system detected a 22-millisecond offset between the event timestamp and the arrival time, contradicting the previously verified time offset resolution fact condition. Based on this, the system revoked the verified time-series resolution fact condition and recorded the transaction event with sequence number 105327 as a conflicting segment. Simultaneously, the system continued to perform short-circuit verification on the remaining resolution fact conditions that had not yet been revoked, without re-verifying the revoked ones.

[0081] Subsequently, the system verified the second condition for the resolution of temporal sequence. Within the remaining time of the disproving window, the system received a total of 7 transaction events in the range of sequence numbers 105328 to 105334, and the sequence intervals of these events all fell within the preset normal range. Based on this, the system determined that the second condition for the resolution of temporal sequence was valid and marked it as verified.

[0082] During the aforementioned proof-of-contrast process, the system consistently adheres to a preset order of irreversible evidence items for verification. When a verified dissolution fact condition contradicts a subsequently arriving fragment, the system promptly revokes the corresponding condition and continues short-circuit verification. Ultimately, since the dissolution fact conditions used to resolve structurally anomalous evidence remain valid throughout the proof-of-contrast window, and at least one temporal dissolution fact condition that does not logically depend on the structurally anomalous evidence is valid, the system determines that the minimum revocation subset corresponding to the anomalous candidate is valid. Therefore, the anomalous candidate is revoked before the proof-of-contrast window expires, and the transaction event is migrated to the submission domain as committable cleaned data for output.

[0083] For the abnormal candidate transaction event with sequence number 105318, the system has determined the verification order of the dissolution fact conditions based on the irreversibility of the evidence items, and has entered the specific verification stage within the disproving evidence window. To avoid frequent revocation of dissolution fact conditions after verification due to short-term fluctuations or local anomalies, the system sets a corresponding commitment lock-in interval for each dissolution fact condition to be verified in the preset order. This interval is used to limit the observation range that the dissolution fact condition needs to maintain consistency before being confirmed. For example... Figure 8 As shown, the length of the committed locking interval is determined by multiple statistical factors, and the contribution ratio of different factors to the final length is presented intuitively.

[0084] When the system begins verifying the factual conditions for resolving structural anomaly evidence, the verification starting point for these factual conditions is determined as the arrival time of the supplementary settlement event with sequence number 105320. A pre-defined observation window W is then constructed in the temporary storage domain centered on this starting point. Observation window W covers the streaming data within the sequence number interval from 105318 to 105330, containing a total of 12 subsequent arrival segments, which serve as the statistical objects for this commitment-locked interval calculation.

[0085] The system calculates various statistics within the observation window W. First, the system calculates the maximum value of the sequence number backwards. Statistically, there is a transaction event within the observation window W that arrived out of order due to network issues, with sequence number 105317. This event backwards from the current data sequence number 105320, with a backwards value of 3. No other segments show a larger backwards. Therefore, there is... Subsequently, the system calculates the arrival time jitter range. For each segment within the observation window W, the system calculates the difference between its arrival time and the event timestamp:

[0086]

[0087] Statistical analysis shows that the time differences between segments within the observation window W range from 2 milliseconds to 18 milliseconds, with a maximum of 18 milliseconds and a minimum of 2 milliseconds. Therefore, the time jitter range is:

[0088]

[0089] Next, the system calculates the proportion of conflicting segments. Of the 12 segments within the observation window W, 2 segments are identified as potentially conflicting with the current resolving factual conditions. Therefore, the conflict proportion can be obtained:

[0090]

[0091] After completing the above statistics, the system calculates the committed locking interval length based on preset parameters. The system's configured time granularity is... milliseconds; preset coefficient value: The system substitutes the above parameters into the formula for calculating the length of the committed locking interval:

[0092]

[0093] get:

[0094]

[0095] Further calculations yield the following:

[0096]

[0097] Combination Figure 8 The contribution ratios shown indicate that the temporal jitter range term accounts for the majority of the committed lock interval length, while the sequence backtracking and conflict ratio terms play a corrective role. Based on this, the system determines the committed lock interval length to be 23, and maps it to a range covering 23 consecutive sequence identifiers starting from sequence number 105320, thus defining the valid verification interval for the dissolution fact condition. Within this committed lock interval, the system continuously monitors whether subsequently arriving segments conflict with the dissolution fact condition.

[0098] Within the committed lock interval, ranging from sequence numbers 105320 to 105343, the system detected no subsequent segments that conflicted with the supplementary settlement event in terms of field consistency, nor any anomalies that could negate the validity of the settlement identifier. Based on this, the system confirmed that no conflicting segments occurred within the committed lock interval, thus confirming that the factual conditions for resolving structurally anomalous evidence had been verified. Subsequently, the system set committed lock intervals for the factual conditions for resolving temporally anomalous evidence in the same manner and performed conflict monitoring within the corresponding intervals.

[0099] As the counter-evidence window entered its second half, the system received a new transaction event at sequence number 105327. For example... Figure 9 As shown, the time offset bar for this transaction event is significantly higher than the threshold line, clearly reflecting the abnormal surge. This transaction event shares the same user identifier as the anomaly candidate and involves an event timestamp field. During the system's verification of this transaction event, it was found that the offset between its arrival time and the event timestamp was 22 milliseconds, significantly higher than the maximum offset of 4 milliseconds previously observed within the committed locking interval, thus directly contradicting the previously verified time-series resolution fact condition. Based on this, the system determined that the time-series resolution fact condition was no longer valid and immediately revoked the previously verified resolution fact condition.

[0100] Upon revoking the factual conditions for resolving the conflict, the system records the transaction event with sequence number 105327 that triggered the conflict as a conflict fragment and writes the conflict fragment to the conflict fragment cache area. The recorded content of the conflict fragment includes at least the transaction event's sequence number, user identifier, event timestamp field, and the offset value of 22 milliseconds that caused the conflict.

[0101] After recording the conflict fragment, the system generates a local masking rule for a preset verification order based on the conflict fragment. To achieve reusability and accurate matching of this rule, the system first calculates a conflict fingerprint for the conflict fragment. The conflict fingerprint includes at least three dimensions of information: first, the sequence identifier range corresponding to the conflict fragment, which is 105325 to 105329 in this embodiment; second, the set of fields involved, which includes the event timestamp field and the arrival time field in this embodiment; and third, the evidence item identifier associated with the conflict fragment, which is the temporal anomaly evidence identifier in this embodiment.

[0102] The system binds conflict fingerprints to local masking rules and applies these rules to the current disproving window and the next two consecutive disproving windows. While these local masking rules are in effect, if the system detects a new disproving window containing a conflicting segment whose sequence identifier range overlaps with 105325 to 105329, involves the same set of fields, and has an evidence item identifier indicating temporal anomalies, the system will skip the verification step altogether, instead of continuing to verify the temporal resolution factual condition corresponding to that evidence item identifier.

[0103] Meanwhile, the system dynamically adjusts the verification order, switching the disproving process to prioritize verifying other disproving factual conditions that can resolve evidence items associated with conflicting fingerprints. In this embodiment, the system refocuses its verification efforts on structural disproving factual conditions that have not yet been revoked and have higher irreversibility, as well as sequence interval disproving factual conditions that are not directly related to conflicting fingerprints, thereby avoiding repeated attempts at verification paths that have been proven unstable under the same conflict mode. Through the above mechanism, even if the system receives a transaction event with a large time offset again in the subsequent disproving window, it will not repeatedly trigger invalid verification of the same temporal disproving factual condition. Instead, it achieves adaptive optimization of the verification order through the synergistic effect of local masking rules and conflicting fingerprints. Finally, before the disproving window ends, the system confirms that the disproving factual conditions used to resolve structurally anomalous evidence have never been overturned, and at least one other temporal disproving factual condition is met, thereby determining that the corresponding minimum revocation subset is met, successfully revoking the anomalous candidate and migrating the transaction event to the submission field output.

[0104] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A real-time streaming data cleaning and anomaly detection method, characterized in that... include: The input streaming data is assigned a timestamp and a sequence number and written to a temporary storage field to form a playback segment of a preset duration. Perform consistency and time sequence legality checks on the temporary data, generate an abnormal evidence set, and mark the data as an abnormal candidate and record the evidence if the triggering conditions are met. Based on the evidence, generate the counter-evidence conditions and open the counter-evidence window for the abnormal candidate. Use subsequent related fragments to verify the counter-evidence conditions. If the counter-evidence conditions are met within the counter-evidence window, cancel the abnormal candidate and migrate it to the submission domain as clean data that can be submitted. If the conditions are not met and timeout occurs, confirm the abnormality and output it to the abnormal channel, while retaining the evidence and verification results.

2. The real-time streaming data cleaning and anomaly detection method according to claim 1, characterized in that... The abnormal evidence set is divided into rebuttable evidence and non-rebuttable evidence. Non-rebuttable evidence includes at least evidence of structural damage that cannot be uniquely recovered from redundant fields within the same data, or evidence that forms an irreversible conflict with the data that has been migrated to the submission domain in terms of sequence identifier. When the abnormal evidence set contains non-rebuttable evidence, the abnormal candidate is directly confirmed as abnormal data and output to the abnormal channel, and the rebuttal window is not enabled.

3. The real-time streaming data cleaning and anomaly detection method according to claim 1, characterized in that... The generation of the counter-evidence conditions includes generating corresponding dissolution fact conditions for each evidence item in the abnormal evidence set, and forming at least one minimum revocation subset; performing short-circuit verification on the minimum revocation subset in a preset order within the counter-evidence window, revoking the abnormal candidate when any minimum revocation subset is established, and migrating the corresponding data to the submission field output as submitable cleaned data.

4. The real-time streaming data cleaning and anomaly detection method according to claim 1, characterized in that... After marking the abnormal candidates and recording the abnormal evidence set, evidence locking is performed to keep the abnormal evidence set and the disproving conditions unchanged during the disproving window. New evidence records appearing in the counter-evidence window are considered incremental evidence and are not included in the set of abnormal evidence.

5. The real-time streaming data cleaning and anomaly detection method according to claim 3, characterized in that... The formation of the minimum revocation subset includes performing dependency constraint checks on the evidence items in the abnormal evidence set, eliminating redundant evidence items that are logically implied by other evidence items, and retaining only the dissolution fact conditions corresponding to the evidence items that are not mutually implied, so as to generate at least one minimum revocation subset without redundant conditions.

6. The real-time streaming data cleaning and anomaly detection method according to claim 3, characterized in that... The preset order is arranged according to the irreversibility of the evidence items. First, the dissolution fact conditions that can uniquely dissolve structural evidence are verified, and then the dissolution fact conditions used to dissolve temporal evidence are verified. When a dissolution fact condition that has been verified within the disproving window contradicts the subsequently arriving fragment, the verified dissolution fact condition is revoked and short-circuit verification continues until any minimum revocation subset is established or the disproving window times out.

7. The real-time streaming data cleaning and anomaly detection method according to claim 3, characterized in that... The subsequent related fragments obtained within the counter-evidence window are limited to fragments that have the same sequence identifier as the abnormal candidate or are in a preset sequence range and involve the corresponding field of the abnormal evidence set; when any minimum revocation subset is established and the abnormal candidate is revoked, the data migrated to the submission field output carries the revocation basis identifier, and the revocation basis identifier includes at least the minimum revocation subset identifier that is satisfied and the corresponding resolution fact condition identifier.

8. The real-time streaming data cleaning and anomaly detection method according to claim 6, characterized in that... In the preset order, a corresponding commitment lock interval is set for each dissolution fact condition to be verified. The commitment lock interval is limited by the sequence identifier range or timestamp range on which the dissolution fact condition depends. The dissolution fact condition is confirmed as verified only when no subsequent arriving segment that conflicts with the dissolution fact condition appears within the commitment lock interval. If a conflicting segment occurs within the committed locking interval, the dissolution fact condition will be determined as failed and the short-circuit verification will continue.

9. The real-time streaming data cleaning and anomaly detection method according to claim 6, characterized in that... When the verified factual condition for resolving conflict contradicts the subsequent arriving fragment and is revoked, the subsequent arriving fragment is recorded as a conflicting fragment, and a local masking rule for the preset order is generated based on the conflicting fragment, so that within the same counter-evidence window or a subsequent preset number of counter-evidence windows, the factual condition for resolving conflicting fragments associated with the evidence items is verified first.

10. The real-time streaming data cleaning and anomaly detection method according to claim 9, characterized in that... When generating the local masking rule, a conflict fingerprint is calculated for the conflict fragment. The conflict fingerprint includes at least the sequence identifier range of the conflict fragment, the set of fields involved, and the corresponding evidence item identifier. When a conflict fragment matching the conflict fingerprint appears in the subsequent disproving window, the verification of the dissolution fact condition corresponding to the evidence item identifier is skipped, and the verification order is switched to prioritize the verification of the dissolution fact condition used to dissolve the evidence item associated with the conflict fingerprint.

Images