A Flink-based intelligent data stream processing system and its implementation method

Abstract

The application discloses an intelligent data stream processing system based on Flink and an implementation method thereof, and belongs to the technical field of distributed stream data processing, and comprises the following steps: a data source access layer is used for uniformly accessing multi-source data; an entity monitoring layer is used for capturing business entity changes based on JPA in a non-invasive manner, generating a standardized message body and asynchronously delivering the standardized message body to a double channel; a unified event model layer is used for mapping the standardized message body into a standardized event and expanding the standardized event; an Flink stream processing engine layer is used for complex event processing, state management and window calculation, and supports declarative pipeline definition and dynamic construction; an exception detection layer is used for multi-dimensional exception detection; a distributed task scheduling layer is used for scheduling tasks according to a DAG model, and load balancing is realized through hot spot splitting and state migration; and a result storage and visualization layer is used for storing data by adopting a hot and cold separation strategy and providing a dynamic visualization interface. The application significantly improves the real-time performance, consistency and intelligent level of data processing, and reduces development and maintenance costs.

Description

Technical Field

[0001] This invention belongs to the field of distributed streaming data processing technology, and particularly relates to an intelligent data stream processing system based on Flink and its implementation method. Background Technology

[0002] With the deepening of enterprise informatization and digital transformation, the amount of data generated by business systems is growing explosively. Traditional enterprise-level data processing systems mainly adopt batch processing mode, which cannot meet the needs of real-time data processing. Although streaming processing technology has made some progress in recent years, existing enterprise-level data flow systems still face the following technical challenges in practical applications:

[0003] Challenges of accessing heterogeneous data from multiple sources: The data formats and structures of multiple business systems within an enterprise are different, making data integration difficult. Existing systems need to develop special access solutions for each data source, resulting in high development and maintenance costs.

[0004] Challenges in ensuring real-time performance and consistency: Traditional data processing often uses batch processing or timed polling mechanisms, which are difficult to meet real-time requirements; simple streaming processing is difficult to guarantee data consistency in a distributed environment, especially in the event of system failure.

[0005] Challenges in monitoring changes to business entities: Enterprise systems require timely capture and processing of state changes of numerous business entities. Existing systems often employ intrusive code implementations and lack a unified mechanism for monitoring and processing changes.

[0006] Insufficient intelligence: Existing systems mainly focus on data transmission and simple processing, lacking intelligent data analysis and anomaly detection capabilities, and cannot effectively support the data processing needs of complex business scenarios.

[0007] Insufficient scalability and fault tolerance: As the amount of data increases, system performance can easily become a bottleneck; the fault recovery mechanism in a distributed environment is not perfect, which affects the reliability of the system.

[0008] High development complexity: Developers need to pay attention to low-level details, such as state management and fault recovery, which increases the difficulty of development and maintenance costs.

[0009] While existing stream processing frameworks such as Apache Storm and Apache Spark Streaming offer some stream processing capabilities, they have limitations in areas such as state management, exact-once semantic guarantees, and complex event handling, and cannot meet the comprehensive needs of enterprise applications.

[0010] Therefore, there is a great need for an intelligent data stream processing system based on an advanced stream processing framework, which can achieve unified access, real-time intelligent processing, and distributed high-availability deployment of multi-source heterogeneous data, while reducing development complexity and improving system maintainability and scalability. Summary of the Invention

[0011] To address the aforementioned technical problems, this invention proposes an intelligent data stream processing system based on Flink and its implementation method, thereby resolving the issues present in the prior art.

[0012] To achieve the above objectives, the present invention provides an intelligent data stream processing system based on Flink.

[0013] Includes: a data source access layer, configured with pluggable connectors for connecting multiple types of data sources, enabling unified data access and standardized conversion;

[0014] The entity listening layer, based on the JPA entity listening mechanism, non-intrusively and automatically captures business entity change events and generates standardized message bodies, which are asynchronously delivered to the dual channels; where JPA stands for Java Persistent Application Programming Interface.

[0015] The unified event model layer is used to generate standardized events based on the standardized message bodies mapped by the data source access layer, and then expand and route them to the Flink stream processing engine layer.

[0016] The Flink stream processing engine layer is used for complex event processing, state management, window computation, stream table transformation, and declarative pipeline definition and dynamic construction. Flink is the Apache Flink stream processing framework.

[0017] The anomaly detection layer is used to detect anomalies in data streams based on rule engines, statistical models, and machine learning algorithms.

[0018] The distributed task scheduling layer schedules tasks based on the DAG dependency model and achieves load balancing through hotspot splitting and state transition; where DAG stands for Directed Acyclic Graph.

[0019] The results storage and visualization layer is used to store data using a hot / cold separation strategy and provides a dynamic visualization interface.

[0020] This invention also provides a method for implementing intelligent data stream processing based on Flink, including:

[0021] The JPA entity listener intercepts business entity changes through lifecycle annotations, triggers a recursive lazy loading algorithm to generate snapshots before and after the change, and then parses the @FieldMetadata annotation to filter sensitive fields. Finally, it binds a unique identifier for the business entity change event, which is a combination of the entity primary key, class name, and SnowFlake algorithm, to obtain a standardized message body. Here, @FieldMetadata is the field metadata annotation.

[0022] Standardized message bodies are pushed in parallel to the business queue for downstream service consumption through a dedicated thread pool, and pushed to the Data-Flow queue for processing by the Flink stream processing engine layer.

[0023] Data-Flow queue events trigger the Flink engine to dynamically build the JobGraph pipeline configured by DSL. During execution, CEP complex event detection and rolling window aggregation operations are performed simultaneously. The window aggregation operation calls the RocksDB state backend to store the results and activates the TTL cleanup mechanism. At the same time, the load indicators are monitored in real time to dynamically adjust the TaskManager resource allocation. The TaskManager is the smallest execution unit of a Flink job.

[0024] During stream processing, the Checkpoint snapshot persistence operator state is executed periodically. If a backpressure signal is detected, the Kafka source ingestion rate is adjusted in reverse.

[0025] The system integrates the detection results of the CEP engine with the window aggregation results, inputs them into the rule engine, statistical model, and machine learning detector, and calculates anomaly scores based on the context library. Then, it triggers real-time blocking, manual review, or risk marking according to the graded thresholds. The handling results are fed back to the rule library and model optimization module to achieve closed-loop learning.

[0026] Compared with the prior art, the present invention has the following advantages and technical effects:

[0027] This invention designs a flexible and scalable event model and connector architecture to achieve unified access to multi-source heterogeneous data; based on the JPA entity lifecycle monitoring mechanism, it automatically captures entity change events without modifying business code, significantly reducing system coupling; the innovative dual-channel architecture and end-to-end TraceId tracing solve the problems of data tracing and idempotent processing in a distributed environment; and it integrates a rule engine, statistical model, and machine learning algorithm to form a complete anomaly detection system, significantly improving the intelligence level of data analysis. Attached Figure Description

[0028] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0029] Figure 1 This is a schematic diagram of the overall system architecture according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the data stream processing flow according to an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the distributed task scheduling mechanism according to an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the state management mechanism according to an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of the anomaly detection process according to an embodiment of the present invention;

[0034] Figure 6 This is a schematic diagram of the business document flow interface according to an embodiment of the present invention. Detailed Implementation

[0035] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0037] Example 1

[0038] like Figure 1 As shown, this embodiment provides an intelligent data stream processing system based on Flink, which adopts a hierarchical design. The specific design of each layer is as follows:

[0039] (a) Data source access layer, responsible for connecting various data sources to realize data collection and preprocessing; designed as follows:

[0040] The data source access layer adopts a plug-in design, supporting unified access to multiple data sources. The main components include:

[0041] SourceConnector: Provides a standardized data source connection interface, supporting various data sources such as relational databases, message queues, and file systems;

[0042] Data Transformer: Converts raw data into the system's internal standard format;

[0043] MetadataManager: Manages the metadata information of the data source and supports dynamic configuration and management of the data source.

[0044] The core interface design of the data source connector adopts a standardized interface definition, which includes three core methods: initializing the connector, creating a Flink data source, and closing the connector, to achieve a unified abstraction of different types of data sources.

[0045] (ii) Entity Listening Layer: Based on the JPA entity listening mechanism, the entity listening layer automatically captures change events of business entities, converts them into standardized messages and pushes them to the message queue, thereby achieving non-intrusive data-driven operation.

[0046] The core implementation of entity listening is the DataDrivenEntityListener class, which captures the add, modify, and delete operations of entities through JPA entity listening annotations (@PostPersist, @PostUpdate, @PostRemove) and then sends the change events to the message queue.

[0047] The entity listening layer adopts a non-intrusive data-driven design, and its core technical ideas and key technical means are as follows:

[0048] (1) Non-intrusive event capture mechanism;

[0049] JPA provides lifecycle callback annotations such as @PostPersist, @PostUpdate, and @PostRemove to uniformly annotate and listen to logic for all persistent entities, eliminating the need to explicitly write event sending statements in the business code;

[0050] This mechanism utilizes JPA's proxy object interception function to detect changes in entity state before a transaction is successfully committed, ensuring the real-time nature and consistency of "add, modify, and delete" events.

[0051] (2) Algorithm for consistent snapshots before and after changes;

[0052] The monitoring logic first queries the "entity before the change" as the old snapshot in the read-only transaction based on the primary key, and simultaneously holds the "new snapshot" after this persistence operation. To ensure the integrity of the snapshot fields, a recursive lazy loading initialization algorithm is proposed, described as follows:

[0053] Use the Hibernate API to determine whether an object or collection has been initialized;

[0054] All lazy-loaded fields are loaded recursively using a depth-first approach, and then set... <object>Deduplication avoids circular references;

[0055] The maximum recursion depth is limited by maxCompareDepth, balancing integrity and performance.

[0056] The final data snapshot satisfies the three characteristics of "consistency + completeness + no circular dependencies", laying the foundation for subsequent difference calculation and audit traceability.

[0057] (3) Field difference filtering strategy;

[0058] We introduce the self-developed annotation @FieldMetadata and the utility class FieldMetadataResolver to mark fields that "do not need to be included in the stream" (such as large fields, static configuration fields, etc.) at the entity class metadata level.

[0059] The listener uses reflection to convert the fields to be excluded into path expressions (e.g., src.attachments, dist.attachments[*].content) and writes them into the excludeFields collection;

[0060] Two-dimensional prefix filtering enables precise exclusion at the serialization level, ensuring message size reduction while preventing the leakage of sensitive fields.

[0061] (4) Globally unique TraceId generation algorithm;

[0062] A 64-bit globally unique TraceId is formed by combining the Twitter SnowFlake algorithm with the application instance ID. Specifically, TraceId = <entity primary key> + "_" + <entity class> + "_" + ... <snowflakeid>.

[0063] By leveraging the consistent use of TraceId in MQ-Topic, Flink state storage, and audit logs, end-to-end link tracing is achieved through a unified identifier transmission mechanism.

[0064] Using TraceId, generated by concatenating 'entity primary key + SnowFlakeId', as a unique identifier across the entire chain, end-to-end tracing and idempotent control are supported through a global ID matching mechanism.

[0065] (5) Asynchronous high-throughput message delivery mechanism;

[0066] Listeners submit asynchronous tasks through a dedicated "data-driven" thread pool provided by the thread pool factory (ThreadPoolExecutorFactoryBean);

[0067] This mechanism employs a dual-channel delivery strategy:

[0068] 1. aSyncSendMQ → Business queue (used for decoupling downstream microservices);

[0069] 2. aSyncDataFlowMQ → Flink Data-Flow queue (entering the real-time stream computing chain);

[0070] The thread pool parameters (core thread count, queue size) are dynamically adjusted by the business TPS through YAML configuration, ensuring smooth writing to the MQ even during extreme peak periods and avoiding blocking of business transactions.

[0071] (6) Exactly-Once semantics guarantees the role and correlation of data change → event capture → standardized message → real-time stream processing in the entire chain. Through this mechanism, the system can still maintain data consistency in high-concurrency scenarios. Even in simulated network failure and node crash scenarios, the data loss rate is reduced to below 0.001%, which improves the data accuracy by 99.9% compared with the traditional At-Least-Once semantic message processing mechanism.

[0072] In the "standardized message → real-time stream processing" stage, the MQ adopts a "transactional message + idempotent consumption" mechanism to ensure that messages are not lost or duplicated during transmission; in conjunction with Flink Checkpoint's snapshot playback mechanism, it ensures data processing consistency during the stream processing stage.

[0073] Logical connections with other hierarchical designs:

[0074] Associated with the entity listening layer: The entity listening logic and the main business transaction belong to the same database transaction domain, ensuring that the message is sent only after the transaction is successfully committed;

[0075] Associated with the unified event model layer: message deduplication and idempotent processing are achieved through the TraceId mechanism;

[0076] Associated with the Flink stream processing engine layer: Utilizes the Checkpoint mechanism to ensure state consistency;

[0077] Associated with the result storage layer: Ensures consistent writing of processing results.

[0078] The exact-once semantic guarantee ensures that even in the event of network failures, node crashes, or other anomalies in a distributed environment, data can be processed exactly once; at the same time, it prevents data loss and duplicate processing, ensuring the accuracy and consistency of business data.

[0079] (7) Scalable and pluggable framework design;

[0080] In the "Event Capture → Standardized Message" stage, the variable parts of the listening logic (MQ sender, field parser, etc.) are injected in the form of Spring Beans, supporting replacement and extension; the DataDrivenSender interface allows access to various message middleware, improving the system's cross-platform adaptability.

[0081] Logical connections with other hierarchical designs:

[0082] Associated with the data source access layer: Supports unified access to multiple data sources through standardized interface definitions;

[0083] Associated with the unified event model layer: Supports the dynamic addition of new event types through scalable event model and schema management;

[0084] Associated with the Flink stream processing engine layer: Supports flexible definition of processing logic through DSL configuration and dynamic pipeline construction;

[0085] Associated with the anomaly detection layer: Supports plug-in integration of multiple detection algorithms.

[0086] By using a scalable and pluggable framework design, the coupling between system components is reduced, the maintainability and scalability of the system are improved, the rapid adaptation to business scenarios and the smooth upgrade of technical components are supported, and the architectural foundation is provided for the long-term evolution and technological iteration of the system.

[0087] Through the above technical solution, this invention achieves automatic connection of the entire chain of "data change → event capture → standardized message → real-time stream processing" without modifying the business service code, which significantly improves the real-time performance, maintainability and intelligence level of enterprise applications.

[0088] Business entity classes need to be annotated with `@EntityListeners`, specifying `DataDrivenEntityListener` as the entity listener. When a business entity changes, `DataDrivenEntityListener` will capture the change event, obtain the entity's state before and after the change, construct a standardized message body (`DataDrivenMqBody`), and then asynchronously send it to the message queue. The standardized message body includes information such as `traceId` (a unique identifier for the change event), `queue` (the name of the message queue), `src` (the entity object before the change), and `dist` (the entity object after the change).

[0089] (III) Unified event model layer design;

[0090] The unified event model layer serves as a bridge connecting the entity listening layer and the Flink stream processing engine layer. It transforms data from different sources and formats into a unified event model for easier subsequent processing. The core of the unified event model is the `BusinessEvent` class, which contains basic event information (such as event ID, event type, event time, and event source) and specific business data. The technical features and implementation methods of the unified event model include:

[0091] (1) Standardized event design;

[0092] The system is designed with a two-tier event structure that includes metadata and business data, as detailed below:

[0093] BusinessEvent{

[0094] / / Metadata section

[0095] String eventId; / / Unique identifier for the event

[0096] String eventType; / / Event type

[0097] long eventTime; / / Event timestamp

[0098] String eventSource; / / Event source

[0099] / / Business data section

[0100] Object businessData; / / Business data object

[0101] }".

[0102] The event type adopts a hierarchical namespace design: {domain}.{subdomain}.{action}, such as order.purchase.created, to ensure global uniqueness; the eventId is automatically consistent with the TraceId mentioned above, realizing the association between event tracking and entity changes.

[0103] (2) Schema dynamic management mechanism;

[0104] It adopts a dual-mode support strategy of JSON Schema + Avro Schema, where JSON Schema is used to flexibly handle changing business data, and Avro Schema is used for high-performance serialization / deserialization;

[0105] By designing the SchemaRegistry component, the schema definitions for all event types can be centrally managed in ZooKeeper;

[0106] Schema evolution is supported by version number marking and compatibility checking mechanisms, and forward and backward compatibility is achieved through field default values ​​and compatibility rules.

[0107] (3) Event transition and routing algorithm;

[0108] Event Conversion: By defining the generic converter interface EventConverter<S,T> It also provides multiple implementation classes to convert data from different sources into a unified BusinessEvent.

[0109] Routing strategy: Use matching rules based on event type and event metadata to determine which Flink processing pipeline a message is routed to.

[0110] Filtering Enhancement: Using the Chain of Responsibility pattern of Filter Chain and Enrichment Chain, events are filtered and enhanced.

[0111] Additional explanation of the event enhancement steps: First, the event transformer converts the standardized message body into an initial BusinessEvent object; then, the event enhancer receives this initial event object; the event enhancer obtains additional context information by querying external data sources (such as databases, caches, APIs, etc.); the obtained context information is added to the properties of the event object to form the enhanced event; finally, the enhanced event is passed to the event router for further processing.

[0112] The event enhancer uses the chain of responsibility pattern to implement multi-level enhancement processing; it supports flexible enhancement strategies through a configurable enhancer chain; each enhancer focuses on adding specific types of contextual information (such as dimensional data, related entities, historical data, etc.).

[0113] (iv) Flink stream processing engine layer design;

[0114] The Flink stream processing engine layer is the core processing component of this system. Based on Apache Flink, it implements powerful stream processing capabilities, supporting complex event handling, state management, window computation, and stream table transformation. The technical features and implementation methods of the Flink stream processing engine layer are as follows:

[0115] (1) Dynamic pipeline construction framework;

[0116] The system uses a declarative pipeline definition DSL to describe the stream processing flow in YAML or JSON format. Based on the DSL configuration, the system calls the FlowBuilder interface to parse the configuration content and dynamically build and deploy the Flink JobGraph. The system adopts a component-based design pattern to realize the hot-swappable function of pipeline components, which allows individual processing components to be dynamically updated without stopping the entire process.

[0117] (2) Distributed state management and fault tolerance mechanism;

[0118] It utilizes Flink's State Backend for efficient state management, supporting three state storage methods: memory, file system, and RocksDB.

[0119] A two-layer state management strategy was designed: hot data is stored in the Flink state for real-time processing; cold data is periodically stored in an external database for historical queries. The state manager, as the interface component between the Flink stream processing engine layer and the state management layer, is responsible for calling the state storage, retrieval, and management functions provided by the state management layer.

[0120] By integrating Flink's Checkpoint mechanism and extending the Savepoint functionality, a two-layer fault-tolerant system is built. When a system failure occurs, the processing state is automatically restored using the most recent checkpoint data, ensuring data consistency during fault recovery.

[0121] By configuring the TTL (Time-To-Live) parameter of the Flink state backend and combining it with a custom cleanup strategy, state data that has not been accessed for a long time can be automatically marked and cleaned up, thereby avoiding the state bloat problem and improving the long-term stability of the system.

[0122] (3) Complex Event Processing Engine (CEP Engine);

[0123] A complex event processing engine based on Flink CEP is implemented, supporting event pattern definition and detection; a visual pattern editor is designed, allowing business users to define complex event patterns through drag-and-drop; multi-dimensional event association algorithms are implemented, such as TraceId association, time window association, entity ID association, etc.; and the definition and detection of positive patterns (such as event sequences) and negative patterns (such as timeout detection) are supported.

[0124] (4) Adaptive flow control and backpressure mechanism;

[0125] By monitoring the backpressure metric of Flink tasks and combining it with a custom threshold control algorithm, the system automatically adjusts the ingestion rate of the upstream data source when backpressure is detected in the downstream processing components, thereby achieving dynamic load balancing and ensuring processing stability. It also implements multi-level caching and batch processing optimization to balance throughput and latency, and provides configurable data sampling and aggregation strategies to ensure system stability in high-traffic scenarios.

[0126] The Flink stream processing engine layer can be divided into pipeline construction unit, event pattern detection unit, state calculation unit, data augmentation unit, and fault tolerance unit;

[0127] The pipeline building unit is used to parse the DSL configuration and trigger the FlowBuilder to dynamically build the JobGraph. The DSL configuration is the configuration of the Flink job, the FlowBuilder is the flow builder interface, and the JobGraph is the job topology graph.

[0128] The event pattern detection unit is used to scan the event stream through the CEP engine, match the preset complex patterns, and trigger alarms or aggregation operations. The CEP engine is a complex event processing engine.

[0129] The state calculation unit is used to call the RocksDB state backend storage for intermediate results and synchronously activate TTL to clean up expired states when performing window aggregation; where TTL is the time-to-live mechanism; RocksDB is an embedded key-value storage database;

[0130] Data augmentation unit, used to associate external dimension table extended event attributes;

[0131] The fault-tolerant unit ensures data consistency during fault recovery through fault-tolerant mechanisms based on Checkpoint and Savepoint. Checkpoint and Savepoint are both mechanisms in the Flink stream processing framework. Checkpoint is the automatic periodic saving of the state of stream processing jobs in Flink, while Savepoint is the manual triggering of saving the state of stream processing jobs in Flink.

[0132] (v) State Management Layer: This layer manages the state information during stream processing, ensuring data consistency during fault recovery. State management is one of the core capabilities of the Flink stream processing engine layer. The functions contained in this layer are called through the state manager. The technical features and implementation methods of this layer are as follows:

[0133] (1) Multi-level state storage architecture:

[0134] A multi-level state storage architecture was designed, comprising memory, RocksDB, and external storage. The appropriate storage level was selected based on the frequency and importance of data access. Hot state data is stored in memory to support high-frequency, low-latency access; large-scale state data is stored in RocksDB to balance performance and capacity; and historical state data is periodically archived to the external storage system for historical querying and analysis.

[0135] (2) State lifecycle management:

[0136] An innovative state TTL (Time-To-Live) mechanism is introduced to set a reasonable lifecycle for each type of state data; a dual TTL strategy based on access time and update time is implemented to more accurately control the lifecycle of state data; a state cleanup scheduler is designed to periodically trigger the cleanup of expired states to avoid memory pressure and performance degradation caused by the infinite growth of state data; incremental cleanup of state data is supported to avoid the cleanup process from impacting the performance of real-time processing.

[0137] (3) Distributed state consistency guarantee:

[0138] By integrating Flink's Checkpoint mechanism and extending its triggering conditions, a consistent snapshot of the distributed state is achieved, supporting state reconstruction during fault recovery. The Checkpoint triggering strategy and storage method are optimized using adaptive triggering frequency and hierarchical storage to balance consistency guarantees and performance overhead. An incremental Checkpoint mechanism is implemented using Change Data Capture (CDC) technology, persisting only changed state data to reduce Checkpoint overhead. A state version management mechanism is designed with dual timestamps and version numbers to support state rollback and historical state queries.

[0139] (4) Status query and access optimization:

[0140] An efficient state indexing mechanism is implemented by combining B+ trees and inverted indexes, supporting multi-dimensional queries of state data by primary key, time range, attribute value, etc. A multi-level caching strategy is used to design a state cache layer to cache frequently accessed state data and reduce query latency. An asynchronous state access API is provided based on the Future / Promise pattern to avoid synchronous queries blocking the main processing flow. Vectorized batch processing technology is used to support batch reading and writing of state data, improving throughput.

[0141] (5) State transition and load balancing:

[0142] A key group-based state partitioning strategy was implemented to support uniform distribution of state data; a state hotspot detection mechanism was designed to identify uneven state access; online migration of state data was supported to achieve dynamic load rebalancing; and data transmission and synchronization strategies during state migration were optimized to minimize the impact of migration on real-time processing.

[0143] The state management module is tightly integrated with the Flink stream processing engine, providing powerful state support for functions such as complex event processing, window calculation, and pattern detection. At the same time, through innovative state lifecycle management and distributed consistency guarantee mechanisms, it effectively solves the problem of state bloat caused by long-term operation, thereby improving the stability and reliability of the system.

[0144] Figure 4 It demonstrates the layered design of state management and the data flow. The layered design includes keyed states (value states, list states, and mapped states), multiple state backend implementations (memory state backend, file system state backend, and RocksDB state backend), checkpoint mechanism, state recovery process, and the generation and transformation of computed states.

[0145] (vi) Anomaly detection layer design;

[0146] The anomaly detection layer, based on various detection strategies and algorithms, identifies and processes abnormal patterns in the data stream. It integrates a rule engine, statistical analysis, and machine learning models to form a multi-layered, comprehensive anomaly detection system. The technical characteristics and implementation methods of anomaly detection are as follows:

[0147] (1) Multidimensional anomaly detection strategy;

[0148] Rule Engine Detection: Anomaly detection is performed based on predefined business rules. The rule execution engine is implemented by integrating the Drools rule language and execution engine API, supporting complex conditional expressions and rule chains. The rule file dynamic loader and version control mechanism enable rules to support dynamic loading and hot updates without restarting the stream processing job.

[0149] Statistical model detection: Based on statistical methods, data is modeled and anomaly detection is performed. Various statistical methods, including Z-Score, MAD (median absolute deviation), and EWMA (exponential weighted moving average), are implemented through sliding window calculation and probability distribution fitting techniques. Historical data analysis and seasonal adjustment algorithms are used to support adaptive threshold adjustment and dynamically set anomaly judgment criteria according to data distribution characteristics.

[0150] Machine learning detection: It utilizes machine learning algorithms for complex pattern recognition and anomaly detection, and supports multiple anomaly detection algorithms through feature engineering and model encapsulation techniques; it adopts a model training strategy that combines online and offline learning by using incremental learning and batch training methods, and supports continuous model optimization through model evaluation and automatic parameter tuning mechanisms.

[0151] (2) Context-aware anomaly assessment mechanism;

[0152] An exception context repository was designed to store context information such as the time, environment, and related events of the exception occurrence; a context-based exception scoring algorithm was implemented, which comprehensively considers the results of multiple detectors and historical information; and a business weight matrix was introduced to assign different weights to different types of exceptions according to their business importance.

[0153] The specific allocation rules for the business weight matrix and the mechanism for obtaining business importance are as follows:

[0154] 1) Methods for obtaining business importance:

[0155] The system obtains basic business weights through business domain configuration files maintained by the configuration center; it supports business experts to dynamically adjust weight values ​​based on actual business conditions through the management interface; and it automatically optimizes weight configurations by combining historical anomaly impact analysis data.

[0156] 2) Weighting rules:

[0157] The system uses a two-dimensional matrix structure of business domains and exception types to store weight relationships. Core business processes (such as payment transactions and order confirmations) are assigned weights ranging from 0.8 to 1.0, while non-core business processes (such as log recording and statistical analysis) are assigned weights ranging from 0.1 to 0.5. The system dynamically adjusts the final weights based on time factors (such as a weighting coefficient of 1.5 during peak business periods and 1.0 during off-peak periods) and contextual information (such as the exception impact range coefficient: 1.5 for global impact and 1.0 for local impact).

[0158] Weight calculation formula: Final weight = Base weight × Time factor × Influence factor;

[0159] This business semantic-based weighting mechanism ensures consistency between anomaly detection results and actual business priorities, effectively improving the targeting and value of anomaly handling.

[0160] (3) Self-learning exception pattern library;

[0161] An anomaly pattern repository was designed to store discovered anomaly patterns and their features. A user feedback-based pattern learning mechanism was implemented through user feedback collection and weight adjustment algorithms to continuously optimize anomaly detection performance. A vector space model and cosine similarity calculation were used to support similar pattern search, and new anomalies similar to historical anomalies were quickly identified through similarity calculation.

[0162] (4) Anomaly warning and intervention mechanism;

[0163] A multi-level early warning mechanism is implemented through anomaly scoring and classification and notification channel selection strategies, issuing different levels of alarms based on the severity of the anomaly; preventive intervention strategies are designed using trend prediction and threshold early warning technologies to provide risk warnings and guidance on preventive measures before anomalies occur; and a processing flow that combines automated response and manual intervention is supported by a combination of rule-driven automated operation and manual confirmation processes.

[0164] The anomaly detection layer includes a multi-dimensional parallel detection unit, a context-aware evaluation unit, and an early warning and response unit;

[0165] The multi-dimensional parallel detection unit is used to obtain rule matching results, indicator deviation scores and anomaly probability scores, and input them into the context-aware evaluation unit;

[0166] The context-aware evaluation unit is used to obtain anomaly scores with context labels based on rule matching results, indicator deviation scores, and anomaly probability scores.

[0167] The early warning and response unit performs graded early warning and response based on the anomaly score with context labels, and feeds back the handling results to the anomaly pattern library.

[0168] like Figure 5 As shown, the anomaly detection process in this embodiment of the invention adopts a multi-level detection and processing architecture. The data flow first undergoes feature extraction, and then is divided into three parallel processing paths: a pattern recognition path (including statistical anomaly detection and machine learning models) and a rule calculation path (processed by a rule engine). The results of the three paths converge at the threshold judgment stage to comprehensively evaluate the anomaly situation, and are then uniformly processed by the alarm management module. Anomaly handling is divided into two main categories: notification and automatic response. Notifications include email notifications, SMS notifications, and Webhook callbacks; automatic responses include automatic retries, resource adjustments, and fault recovery. Figure 5 The entire process from data input to anomaly response is clearly demonstrated, and the actual operation mechanism of the above-mentioned anomaly detection technology is presented intuitively. In particular, the key links of multi-dimensional parallel detection and hierarchical response ensure that the system can accurately identify and efficiently handle various anomalies.

[0169] (vii) Design of the distributed task scheduling layer;

[0170] The distributed task scheduling layer is responsible for the intelligent scheduling and resource management of stream processing tasks, ensuring high availability, scalability, and resource utilization efficiency of the system. Based on Flink's distributed architecture, this layer implements intelligent task scheduling and dynamic resource allocation. The technical characteristics and implementation methods of distributed task scheduling are as follows:

[0171] (1) Intelligent resource allocation algorithm;

[0172] A load-aware resource allocation algorithm is implemented through monitoring metric collection and load prediction models to dynamically adjust resource allocation based on task complexity, data volume, and processing time. A resource quota management mechanism is designed using quota limits and priority strategies to rationally allocate computing resources and avoid resource contention. Automatic scaling triggers and resource schedulers are adopted to support elastic scaling, automatically adjusting task parallelism and resource configuration according to traffic changes.

[0173] (2) High-availability cluster management;

[0174] ZooKeeper's ephemeral nodes and listener mechanism enables distributed coordination and leader election, ensuring high availability for cluster management. A multi-level fault detection and recovery strategy is designed using heartbeat detection, timeout judgment, and fault isolation, including process-level, node-level, and cluster-level fault handling. Version control and traffic switching technologies are employed to support rolling upgrades and blue-green deployments, ensuring smooth system upgrades and migrations.

[0175] (3) Task dependency management and scheduling optimization;

[0176] Design a task dependency model based on DAG (Directed Acyclic Graph) to accurately describe the dependencies between tasks; implement topology sorting and critical path analysis algorithms to optimize task execution order; support dynamic scheduling based on data readiness to improve resource utilization efficiency.

[0177] (4) Load balancing and hotspot migration;

[0178] Implement a dynamic rebalancing algorithm for data partitioning to solve the data skew problem; design a hot task detection and splitting mechanism to dynamically decompose high-load tasks into multiple subtasks; support task and state migration to optimize overall resource utilization.

[0179] The distributed task scheduling layer can be divided into a resource awareness unit, an intelligent decision-making unit, an elastic execution unit, and a fault self-healing unit.

[0180] The resource awareness unit is used to report the CPU, memory, back pressure and data skew of each node in real time through the monitor, and aggregate them into a load profile through the resource manager.

[0181] The intelligent decision-making unit is used to call the load awareness algorithm based on the load profile through the task scheduler to calculate the optimal parallelism and resource quota that satisfy the DAG dependency model. If a hot task is found, it will trigger splitting and generate a sub-task topology.

[0182] The elastic execution unit, the cluster manager, uses ZooKeeper to distribute the decisions of the intelligent decision-making unit, adjust resource allocation and migrate status online, and complete the version rolling update. ZooKeeper is a distributed coordination service.

[0183] The fault self-healing unit is used to automatically activate standby backup nodes when a process, node, or cluster-level anomaly occurs through the fault recovery manager, restore the job status based on the latest savepoint, and reschedule tasks in real time.

[0184] like Figure 3 As shown, the distributed task scheduling mechanism of this invention adopts a hierarchical task management architecture, achieving efficient task allocation and resource scheduling. This mechanism consists of a resource manager, a job manager, a checkpoint coordinator, and a task manager, realizing intelligent scheduling, dynamic load balancing, fault self-healing, and elastic scaling based on a DAG dependency model, effectively solving the problems of data skew and hot tasks.

[0185] (viii) Design of results storage and visualization layer;

[0186] The results storage and visualization layer is the presentation layer of this system. It is responsible for storing, querying, and visualizing the results, providing users with intuitive data insights and decision support. This layer employs multi-level storage and flexible visualization design to achieve efficient data management and presentation. The technical features and implementation methods of results storage and visualization are as follows:

[0187] (1) Multi-level storage architecture;

[0188] A high-performance real-time query processing engine is implemented through a query parsing and execution framework, supporting SQL and SQL-like query languages; a query optimizer is designed using execution plan optimization techniques to automatically select the best execution plan; multidimensional analysis and real-time aggregation, such as time dimension, spatial dimension, and entity dimension, are supported based on dimensional indexing methods; and query results are cached through a cache management mechanism to improve query performance.

[0189] The multi-level state storage architecture differs from the state management layer in the following essential ways:

[0190] The difference in data nature is as follows: the result storage layer processes the final business result data after the stream processing is completed, which is oriented towards end users for querying and analysis; while the state management layer processes the intermediate state data in the stream computing process, which is mainly used for calculation recovery and stream processing logic.

[0191] Access mode differences: The data in the result storage layer is mainly for external applications and user queries, and needs to support complex query patterns and high-concurrency reads; the data in the state management layer is mainly for Flink's internal computation process, with efficient state updates and checkpoint generation as the main goals.

[0192] Storage lifecycle differences: Data in the result storage layer is usually stored for a longer period of time and requires complete data lifecycle management; data in the state management layer usually only exists during the execution of the stream processing job and may no longer be needed after the job ends.

[0193] Technical implementation differences: The result storage layer uses application-oriented storage systems such as Redis, ClickHouse, Elasticsearch, and HDFS; the state management layer mainly uses Flink's built-in state backends such as memory and RocksDB, which are storage systems optimized for stream processing.

[0194] The multi-level storage architecture of this layer is specifically implemented as follows: hot data is stored in an in-memory database or distributed cache to support high-frequency access; warm data is stored in a columnar storage system to support fast aggregation queries; cold data is stored in a distributed file system to support large-scale historical data archiving; a data lifecycle management mechanism is designed to automatically complete the cold and hot migration of data; and multiple storage backends are supported, such as Redis, ClickHouse, Elasticsearch, HDFS, etc., to meet the needs of different scenarios.

[0195] (2) Real-time query engine;

[0196] A high-efficiency real-time query processing engine is implemented, supporting SQL and SQL-like query languages; a query optimizer is designed to automatically select the best execution plan; multidimensional analysis and real-time aggregation are supported, such as time dimension, spatial dimension, entity dimension, etc.; a query result caching mechanism is implemented to improve query performance.

[0197] (3) Dynamic visualization framework;

[0198] The component registration and layout management system designs a component-based visualization framework that supports various visualization methods such as charts, dashboards, and heatmaps. It utilizes data feature recognition and display rule matching to realize data-driven visualization components, automatically selecting the appropriate display format based on data characteristics. It adopts event listening and state management mechanisms to support interactive analysis, such as drill-down, filtering, and comparison operations. It provides a visualization template library through a template definition language and rendering engine, facilitating the rapid construction of business dashboards.

[0199] (4) Real-time alarm and notification system;

[0200] A multi-channel alarm push mechanism is designed using message queues and notification adapters, supporting email, SMS, enterprise instant messaging tools, etc. Similarity calculation and time window aggregation are used to implement intelligent alarm grouping and merging algorithms to avoid alarm storms; a time threshold and escalation rule engine is adopted to support an alarm escalation mechanism, automatically escalating alarms based on their duration and severity.

[0201] The result storage and visualization layer can be divided into a result writing unit, a lifecycle migration unit, a real-time query unit, a dynamic visualization unit, and an alarm unit.

[0202] The result writing unit processes each output from the Flink stream processing engine layer through the Sink to obtain the event popularity; the storage manager assigns tags to events based on the event popularity field and performs multi-level storage based on the tags; where Sink is the receiver operator;

[0203] The lifecycle migration unit is used to automatically migrate data when Redis memory usage is high or ClickHouse partition age triggers a threshold. After the migration is completed, the new partition metadata is synchronized to the SchemaRegistry. At the same time, it sends resource release or request signals to the resource manager of the distributed task scheduling layer through the API gateway. Here, Redis is a Redis in-memory database, ClickHouse is a ClickHouse columnar database, and SchemaRegistry is the schema registry center.

[0204] The real-time query unit retrieves the cached query results based on the query request. If no match is found, it uses multidimensional indexes and partition pruning to generate the optimal execution plan, performs parallel scanning and aggregation of multi-level stored data, and returns the aggregation results.

[0205] The dynamic visualization unit receives the aggregated results returned by the real-time query unit and automatically matches the display format from the visualization template library according to the data dimensions and features; the interaction logs generated by user operations are written back to the anomaly pattern library in real time, providing incremental samples for the retraining of the anomaly detection layer model;

[0206] The alarm unit uses a comprehensive anomaly score to invoke an intelligent alarm grouping and merging algorithm for multi-channel, tiered push notifications.

[0207] The core components and responsibilities of the results storage and visualization layer are as follows:

[0208] Storage Manager: Manages multi-level storage systems and implements a hot / cold data separation strategy. Hot data is stored in in-memory databases such as Redis, warm data is stored in columnar storage systems such as ClickHouse, and cold data is stored in distributed file systems such as HDFS. It also designs a data lifecycle management mechanism to automatically complete the migration of data between hot and cold storage.

[0209] Query Engine: Handles various data query requests, supports SQL and SQL-like query languages; designs a query optimizer to automatically select the best execution plan; supports multidimensional analysis and real-time aggregation; implements a query result caching mechanism to improve query performance.

[0210] Visualization Engine: Generates various charts and dashboards. It features a modular visualization framework that supports multiple visualization methods such as charts, dashboards, and heatmaps; implements data-driven visualization components that automatically select the appropriate display format based on data characteristics; supports interactive analysis; and provides a visualization template library.

[0211] Alarm Center: Manages alarm rules and notification policies, designs multi-channel alarm push mechanisms, supports email, SMS, enterprise instant messaging tools, etc.; implements intelligent alarm grouping and merging algorithms; and supports alarm escalation mechanisms.

[0212] API Gateway: Provides a unified data access interface across multiple storage levels, offers RESTful API interfaces, and uniformly encapsulates access to multi-level storage systems; implements access control and rate limiting; and supports data format conversion and protocol adaptation.

[0213] Result writing unit: Write the result of the processing stream.

[0214] The output of the Flink stream processing engine layer is processed by the Sink operator to obtain the event popularity; the event is tagged according to the event popularity, and multi-level storage distribution is carried out through the data writing mechanism in the multi-level storage architecture.

[0215] Lifecycle Migration Unit: Manages the lifecycle of data. When Redis memory usage or ClickHouse partition age triggers a threshold, it automatically performs data migration; after migration, it updates metadata; and sends resource release or request signals to the resource manager.

[0216] Real-time query unit: Performs efficient data queries. Retrieves cached query results; generates optimal execution plans using multidimensional indexes and partition pruning; performs parallel scanning and aggregation of multi-level stored data.

[0217] Dynamic visualization unit: Enables intelligent data visualization.

[0218] The visualization template library automatically matches the display format based on data dimensions and features; user interaction logs are written back to the Pattern Repository in real time, providing incremental samples for the anomaly detection layer.

[0219] Results storage is based on Flink's Sink mechanism, supporting the writing of processing results to various target systems such as Elasticsearch, Kafka, and databases. Visualization is based on web technologies, providing rich charts and interactive analysis functions. This system implements flexible results storage and visualization capabilities, supporting multiple storage methods and visualization displays.

[0220] like Figure 6 As shown, the system's business document flow interface intuitively displays the complete business process from purchase contracts to pre-issued invoices. By transforming complex data processing results into an intuitive business view, it achieves seamless integration of technology and business, providing enterprises with real-time and comprehensive business monitoring capabilities.

[0221] This system achieves seamless collaboration across all levels through innovative technological integration mechanisms, enabling end-to-end coordination from data capture to processing and analysis. The key integration mechanisms are as follows:

[0222] (a) A full-link data tracing mechanism based on TraceId;

[0223] The TraceId generated by the entity listening layer (composed of <entity primary key> + "_" + <entity class> + "_" + ... <snowflakeid>This component (often used throughout the entire processing flow) is retained as an eventId in the unified event model layer, serves as a status key in Flink stream processing, is used for correlation analysis in the anomaly detection layer, and is used for data tracing in the result storage layer. This end-to-end identification mechanism solves the challenges of data tracing and idempotent processing in distributed systems.

[0224] (b) Dual-channel decoupling and coordination mechanism;

[0225] The entity listening layer innovatively designs a dual-channel architecture with parallel business queues and data-flow queues. The former serves to decouple the business system, while the latter is dedicated to real-time stream processing. The two channels are physically isolated but logically unified, sharing the same message body structure and TraceId, thus achieving loose coupling and data consistency between the business system and the real-time analysis system.

[0226] (c) Hierarchical status management and data consistency coordination mechanism;

[0227] The entity listening layer ensures reliable data delivery through transactional messages, the unified event model layer avoids duplicate messages through idempotent processing, the Flink stream processing engine layer guarantees state consistency through checkpointing, and the result storage layer ensures result consistency through atomic writes. This cross-layer collaborative mechanism achieves end-to-end exactly-Once semantics, ensuring data processing accuracy even in the event of network failures or node crashes.

[0228] (d) A cross-level coordination mechanism for dynamic resource allocation;

[0229] The distributed task scheduling layer achieves cross-layer resource coordination by monitoring resource usage at each level: when backpressure is detected in the Flink engine layer, the message sending rate of the entity listening layer can be adjusted upwards; when hot tasks are detected, task splitting and state distribution adjustments are triggered; and when the data volume in the result storage layer surges, more storage resources are dynamically allocated. This cross-layer resource coordination mechanism ensures the stability of the system under load fluctuations.

[0230] (e) Metadata-driven dynamic processing link construction mechanism;

[0231] Information sharing and processing logic selection are achieved between different layers through the transmission and enhancement of metadata. Entity types and operation types captured by the entity listening layer serve as metadata, guiding event transformation and routing in the unified event model layer. Event metadata, in turn, drives the Flink stream processing engine layer to dynamically select processing pipelines. Anomaly detection results are appended as metadata to the processing results, guiding the storage strategy selection in the result storage layer. This metadata-driven mechanism gives the system high flexibility and scalability.

[0232] The system also includes: a metadata management module for managing various metadata information within the system; a security management module for implementing data access control and security auditing; a monitoring management module for monitoring system operating status and performance indicators; and a configuration management module for managing system configurations and parameter settings. The system adopts a microservice architecture, with modules interacting through standard interfaces to achieve a loosely coupled system architecture, supporting independent deployment and expansion of modules.

[0233] like Figure 2 As shown, this system adopts a distributed deployment architecture. The system includes seven core components: data source, data source connector, data converter, Flink engine, state manager, anomaly detector, and result receiver, forming a complete data processing pipeline.

[0234] (a) JPA entity listener implementation:

[0235] By utilizing the JPA entity monitoring mechanism, automatic capture and event sending of business entity changes are achieved. The specific implementation steps are as follows:

[0236] (1) Define the entity listener class (DataDrivenEntityListener):

[0237] Capture entity add, update, and delete events using the @PostPersist, @PostUpdate, and @PostRemove annotations; obtain snapshots of the entity's state before and after the changes in the event handling methods; construct a standardized message body and send it asynchronously to the message queue.

[0238] (2) Entity class integration listener:

[0239] Add the @EntityListeners(DataDrivenEntityListener.class) annotation to the entity class; the entity class needs to implement the BaseEntity interface to provide unique identification and metadata access capabilities.

[0240] (3) Recursive lazy loading initialization:

[0241] For lazy-loaded fields of an entity, recursion is used for initialization to ensure the integrity of the data snapshot; a maximum recursion depth is set to avoid circular references and performance issues; reflection technology is used to identify and process different types of fields.

[0242] (4) Asynchronous message sending:

[0243] A dedicated thread pool is used to asynchronously process message sending to avoid blocking business transactions; a retry mechanism is implemented for messages that fail to be sent to ensure reliable message delivery; message deduplication and idempotency processing are used to avoid the impact of duplicate messages.

[0244] (II) Flink Stream Processing Implementation:

[0245] This implementation leverages Apache Flink to achieve powerful stream processing capabilities, as detailed below:

[0246] (1) Stream processing job configuration:

[0247] Stream processing jobs can be defined using YAML or JSON configuration files, including data sources, operators, states, windows, etc.; dynamic loading and updating of configurations are supported, and processing logic can be adjusted without restarting the job.

[0248] (2) Complex event handling:

[0249] This implementation leverages the Flink CEP framework to define and detect complex event patterns; it supports multiple pattern types, including sequential, cyclic, and conditional patterns; and it implements a custom pattern matching algorithm to improve matching efficiency and accuracy.

[0250] (3) Status Management:

[0251] RocksDB is used as the state backend to support large-scale state data storage; state expiration and cleanup strategies are implemented to prevent state from growing indefinitely; and checkpoint and savepoint mechanisms ensure state consistency and recoverability.

[0252] (4) Window calculation:

[0253] It supports time windows (scrolling windows, sliding windows, session windows) and counting windows; it implements event time processing based on watermarks to handle out-of-order and late data; and it supports multi-dimensional window aggregation, such as by time, entity ID, event type, etc.

[0254] (5) Dynamic rule processing:

[0255] It implements a dynamic loading and execution mechanism for rules, supporting real-time rule updates; rules are described using a DSL (Domain-Specific Language), making them easy for business personnel to understand and write; it supports rule version management and rollback, ensuring the controllability of rule changes.

[0256] (III) Anomaly Detection Implementation:

[0257] This system implements a multi-layered anomaly detection mechanism, as detailed below:

[0258] (1) Rule engine detection:

[0259] The rule execution engine is based on Drools and supports complex conditional expressions. The rules adopt a declarative syntax, which is convenient for business experts to write and maintain. It supports hot deployment and version management of rules, enabling dynamic updates of rules.

[0260] (2) Statistical model testing:

[0261] It implements a sliding window Z-Score algorithm to detect abnormal fluctuations in numerical indicators; uses an EWMA (Exponentially Weighted Moving Average) model to detect trend anomalies in time series data; and supports adaptive thresholds to dynamically adjust anomaly judgment criteria based on historical data.

[0262] (3) Machine learning detection:

[0263] It integrates the Isolation Forest algorithm for anomaly detection in high-dimensional data; implements an anomaly detection model based on an autoencoder, suitable for complex pattern recognition; and supports hybrid online and offline learning to achieve continuous model optimization.

[0264] (4) Multidimensional correlation analysis:

[0265] It uses TraceId to achieve end-to-end correlation and identify abnormal patterns across systems; it uses graph algorithms to analyze related events and discover potential causal relationships; it supports pattern correlation within a time window and identifies time-series related anomalies.

[0266] (iv) Implementation results;

[0267] In practical application at a large manufacturing enterprise, this system has enabled real-time data acquisition and processing from multiple internal business systems (including ERP, MES, SCM, etc.), building a real-time business monitoring and analysis platform. The system processes over 10 million data entries daily, with an anomaly detection accuracy rate exceeding 95%.

[0268] Table 1

[0269]

[0270] Table 1 (continued)

[0271]

[0272] Table 1 (continued from 2)

[0273]

[0274] Table 1 (continued from 3)

[0275]

[0276] Table 1 (continued from 4)

[0277]

[0278] Table 1 (continued from 5)

[0279]

[0280] All English terms and self-named terms used in this invention follow the standard definitions in the following technical specifications and documents, and can be clearly and uniquely explained through the relevant official documents and Table 1.

[0281] (1) Java Language Specification: Oracle official specification, document: https: / / docs.oracle.com / javase / specs / ;

[0282] (2) JPA: Java Persistence API Specification, document: https: / / jakarta.ee / specifications / persistence / ;

[0283] (3) Apache Flink: A distributed stream processing framework. Documentation: https: / / flink.apache.org / docs / ;

[0284] (4) Spring Framework: Java application framework, documentation: https: / / spring.io / projects / spring-framework;

[0285] (5) Hibernate ORM: Object-Relational Mapping Framework, Documentation:

[0286] https: / / hibernate.org / orm / documentation / ;

[0287] (6) Apache Kafka: Distributed stream processing platform, documentation: https: / / kafka.apache.org / documentation / ;

[0288] (7) Redis: In-memory data storage system, documentation: https: / / redis.io / documentation;

[0289] (8) ClickHouse: a columnar database, documentation: https: / / clickhouse.com / docs / ;

[0290] (9) Elasticsearch: Distributed search engine, documentation: https: / / www.elastic.co / guide / ;

[0291] (10) Apache ZooKeeper: Distributed coordination service, documentation: https: / / zookeeper.apache.org / doc / ;

[0292] (11) RocksDB: an embedded key-value database, documentation: https: / / rocksdb.org / docs / ;

[0293] (12) Drools: Business Rules Management System, Documentation:

[0294] https: / / www.drools.org / learn / documentation.html;

[0295] (13) Machine learning algorithms: Isolation Forest, One-Class SVM, etc. Refer to sklearn documentation: https: / / scikit-learn.org / stable / ;

[0296] (14) Statistical methods: Z-Score, MAD, EWMA, etc. refer to standard statistics textbooks;

[0297] (15) Apache RocketMQ: Distributed messaging platform, documentation: https: / / rocketmQ.apache.org / docs / .

[0298] This embodiment also provides a method for implementing intelligent data stream processing based on Flink, including:

[0299] The JPA entity listener intercepts business entity changes through lifecycle annotations, triggers a recursive lazy loading algorithm to generate snapshots before and after the change, and then parses the @FieldMetadata annotation to filter sensitive fields. Finally, it binds a unique identifier for the business entity change event, which is a combination of the entity primary key, class name, and SnowFlake algorithm, to obtain a standardized message body. Here, @FieldMetadata is the field metadata annotation.

[0300] Standardized message bodies are pushed in parallel to the business queue for downstream service consumption through a dedicated thread pool, and pushed to the Data-Flow queue for processing by the Flink stream processing engine layer.

[0301] Data-Flow queue events trigger the Flink engine to dynamically build the JobGraph pipeline configured by DSL. During execution, CEP complex event detection and rolling window aggregation operations are performed simultaneously. The window aggregation operation calls the RocksDB state backend to store the results and activates the TTL cleanup mechanism. At the same time, the load indicators are monitored in real time to dynamically adjust the TaskManager resource allocation. The TaskManager is the smallest execution unit of a Flink job.

[0302] During stream processing, the Checkpoint snapshot persistence operator state is executed periodically. If a backpressure signal is detected, the Kafka source ingestion rate is adjusted in reverse.

[0303] The system integrates the detection results of the CEP engine with the window aggregation results, inputs them into the rule engine, statistical model, and machine learning detector, and calculates anomaly scores based on the context library. Then, it triggers real-time blocking, manual review, or risk marking according to the graded thresholds. The handling results are fed back to the rule library and model optimization module to achieve closed-loop learning.

[0304] This invention enables multi-source data access through a unified event model and pluggable connectors, and automatically captures changes using zero-intrusion data-driven technology. Through dual-channel message queues and full-link tracing technology, it ensures asynchronous decoupling, end-to-end tracing, and exactly-Once semantics in data processing. Based on Flink state management and a multi-level anomaly detection system, it effectively solves the problems of state expansion and anomaly early warning, thereby improving system stability and intelligence.

[0305] At the data access level, this invention achieves unified access and standardized processing of multi-source heterogeneous data, reducing the complexity of data integration; based on JPA entity monitoring, non-intrusive data-driven processing enables real-time capture of data changes without modifying business code; it supports multiple data source types and has good scalability and flexibility.

[0306] In terms of processing efficiency, this invention is based on Flink's stream processing engine, which enables real-time data processing and significantly improves the timeliness of data processing; through distributed task scheduling and resource optimization, it improves the system's processing efficiency and resource utilization; asynchronous message processing and batch optimization reduce system latency and increase throughput.

[0307] In terms of reliability, based on Flink Checkpoint and state management, it achieves exact one-time semantics for data processing, ensuring that data is not lost or duplicated; the distributed high-availability architecture ensures the stability and fault tolerance of the system; and the multi-level fault recovery mechanism improves the system's recovery capability under abnormal conditions.

[0308] At the intelligence level, multi-layered anomaly detection strategies enhance the ability to identify data anomalies; intelligent processing based on rule engines, statistical analysis, and machine learning supports intelligent analysis of complex business scenarios; and a self-learning anomaly pattern library continuously optimizes anomaly detection performance.

[0309] In terms of availability, the flexible visualization framework provides intuitive data insights and analysis capabilities; the unified API interface facilitates integration with other systems; and the automated alarm and notification mechanism improves system availability and operational efficiency.

[0310] In terms of development efficiency, the component-based and plug-in-based design supports rapid development and system expansion; the unified development interface and framework reduce development complexity; and the declarative configuration method reduces the amount of code and improves development efficiency.

[0311] This invention enables unified access to multi-source heterogeneous data, zero-intrusion data-driven operation, and multi-level anomaly detection, significantly improving the system's real-time performance, reliability, and intelligence. Through this invention, enterprises can build a unified real-time data flow and analysis platform, achieving real-time monitoring, status tracking, and intelligent analysis of business data, providing strong support for enterprise decision-making.

[0312] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / snowflakeid> < / snowflakeid> < / object>

Claims

1. A Flink-based intelligent data stream processing system, characterized in that, The application relates to a data processing system. The data source access layer is configured with a pluggable connector for connecting multiple kinds of data sources to realize unified data access and standardized conversion. The entity monitoring layer automatically captures business entity change events and generates a standardized message body based on a JPA entity monitoring mechanism and non-intrusive mode, and the standardized message body is delivered to a double channel in an asynchronous mode; wherein, JPA is a Java persistence application programming interface. After the entity monitoring layer captures the business entity change events based on a JPA life cycle annotation, a recursive delayed loading algorithm is triggered, an uninitialized field is detected through a Hibernate API, associated objects are recursively loaded and circular references are eliminated in a depth-first strategy, and a pre-change snapshot and a post-change snapshot are generated and filtered; a global TraceId is obtained based on a business entity primary key, a business entity class name and a SnowFlake algorithm; a standardized message body is constructed based on the global TraceId and the pre-change snapshot and the post-change snapshot after field filtering, and the standardized message body is delivered to a double channel in an asynchronous mode; wherein, the global TraceId is a unique identifier for tracking a request process in a distributed system, the SnowFlake algorithm is a snowflake algorithm, the Hibernate is a Java object relationship mapping framework, and the API is an application programming interface. The unified event model layer is used for generating a standardized event and performing extension and routing to a Flink stream processing engine layer according to the standardized message body mapped by the data source access layer. The Flink stream processing engine layer is used for complex event processing, state management, window calculation, stream table conversion and declarative pipeline definition and dynamic construction, and Flink is an Apache Flink stream processing framework. The Flink stream processing engine layer comprises a pipeline construction unit, an event mode detection unit, a state calculation unit, a data enhancement unit and a fault tolerance unit. The pipeline construction unit is used for analyzing a DSL configuration, triggering a FlowBuilder interface to dynamically construct a JobGraph, the DSL configuration is a configuration of a Flink job, the FlowBuilder interface is a stream arrangement interface, and the JobGraph is a job topology graph. The event mode detection unit is used for scanning an event stream through a CEP engine, matching a preset complex mode, and triggering an alarm or an aggregation operation; wherein, the CEP engine is a complex event processing engine. The state calculation unit is used for calling a RocksDB state backend storage to store intermediate results when window aggregation is performed, and synchronously activating TTL cleaning of expired states; wherein, TTL is a survival time mechanism, and RocksDB is an embedded key-value storage database. The data enhancement unit is used for extending event attributes by associating external dimension tables. The fault tolerance unit ensures data consistency when a fault is recovered through a checkpoint and savepoint-based fault tolerance mechanism. Checkpoint and Savepoint are mechanisms in the Flink stream processing framework. Checkpoint is a mechanism in Flink that automatically saves the state of a stream processing job at regular intervals, while Savepoint is a mechanism in Flink that manually triggers the saving of the state of a stream processing job. The anomaly detection layer is used to detect anomalies in data streams based on rule engines, statistical models, and machine learning algorithms. The distributed task scheduling layer is based on the DAG dependency model to schedule tasks and achieve load balancing through hot spot splitting and state migration. DAG is a directed acyclic graph. The result storage and visualization layer uses a hot and cold separation strategy to store data and provides a dynamic visualization interface.

2. The Flink-based intelligent data stream processing system according to claim 1, wherein the data source access layer includes a data source connector, a data converter, and a metadata manager. The data source connector is used to provide a standardized data source connection interface. The data source connector uses a standardized interface definition core interface design, which includes initializing the connector, creating a Flink data source, and closing the connector. The data converter is used to convert raw data into a system internal standard format. The metadata manager is used to manage the metadata information of the data source.

3. The Flink-based intelligent data stream processing system according to claim 1, wherein the dual-channel includes a business queue and a Data-Flow queue. The business queue is used for asynchronous consumption by downstream microservices to achieve system decoupling, and the Data-Flow queue is directly connected to the Flink stream processing engine layer to support real-time stream processing and is physically isolated from the business decoupling queue.

4. The Flink-based intelligent data stream processing system according to claim 1, wherein the unified event model layer uses a hierarchical namespace design to define event types and bind dual-mode Schema. The event converter parses the data source characteristics of the standardized message body to generate standardized business events. The event enhancer associates external context to expand the dimensions of the standardized business events. Based on the type and corresponding metadata of the standardized business events, the event router sends the expanded standardized business events to the Flink stream processing engine layer.

5. The Flink-based intelligent data stream processing system according to claim 1, wherein the anomaly detection layer includes a multi-dimensional parallel detection unit, a context-aware evaluation unit, a pre-warning and response unit. The multi-dimensional parallel detection unit is used to obtain rule matching results, index deviation scores, and anomaly probability scores and input them to the context-aware evaluation unit. The context-aware evaluation unit is used to obtain anomaly scores with context labels based on rule matching results, index deviation scores, and anomaly probability scores. The pre-warning and response unit performs hierarchical pre-warning and response based on anomaly scores with context labels and feeds back the disposal results to the anomaly pattern library.

6. The Flink-based intelligent data stream processing system according to claim 1, wherein ​ ​ ​ ​ The distributed task scheduling layer comprises a resource sensing unit, an intelligent decision unit, an elastic execution unit and a fault self-healing unit; The resource sensing unit is configured to report the CPU, memory, back pressure and data skew of each node in real time through a monitor, and aggregate the load portrait through a resource manager; The intelligent decision unit is configured to call a load sensing algorithm based on the load portrait through a task scheduler, calculate the optimal parallelism and resource quota that meet the DAG dependency model, and trigger splitting and generate a sub-task topology if a hot task is found; The elastic execution unit is configured to have a cluster manager issue the decision of the intelligent decision unit through a ZooKeeper, perform online rolling adjustment of resource allocation and migration of state, complete version rolling update, and have the ZooKeeper serve as a distributed coordination service; The fault self-healing unit is configured to activate a standby node on standby in a process, node or cluster level exception through a fault recovery manager, restore the job state based on the latest Savepoint, and reschedule the task in real time.

7. The intelligent data stream processing system based on Flink according to claim 1, wherein The result storage and visualization layer comprises a result writing unit, a life cycle migration unit, a real-time query unit, a dynamic visualization unit and an alarm unit; The result writing unit processes each output of the Flink stream processing engine layer through a Sink to obtain event heat, and stores the event heat in a storage manager to assign a tag to the event and perform multi-level storage according to the tag; the Sink is a receiver operator; The life cycle migration unit is configured to automatically perform data migration when a Redis memory occupation or a ClickHouse partition age trigger threshold is reached, and synchronize new partition metadata to a SchemaRegistry after the migration is completed; meanwhile, the API gateway sends a resource release or application signal to the resource manager of the distributed task scheduling layer; the Redis is a Redis memory database, the ClickHouse is a ClickHouse columnar database, and the SchemaRegistry is a schema registry; The real-time query unit retrieves a query result cache according to a query request, generates an optimal execution plan using multi-dimensional indexing and partition pruning if the query request is not hit, performs parallel scanning and aggregation on multi-level storage data, and returns an aggregated result; The dynamic visualization unit automatically matches a display form from a visualization template library according to data dimensions and features after receiving the aggregated result returned by the real-time query unit; an interactive log generated by user operation is written back to an abnormal pattern library in real time to provide incremental samples for model retraining of the abnormal detection layer; The alarm unit calls an alarm intelligent grouping and merging algorithm according to a comprehensive abnormal score to perform multi-channel hierarchical push.

8. A Flink-based intelligent data stream processing implementation method, characterized in that, The system comprises: Intercept business entity changes through the life cycle annotation of JPA entity listener, trigger recursive lazy loading algorithm to generate pre-change snapshot and post-change snapshot, then filter sensitive fields after analyzing @FieldMetadata annotation, bind the unique identifier of business entity change event which is spliced by entity primary key, class name and SnowFlake algorithm, obtain standardized message body, wherein, @FieldMetadata is field metadata annotation; Push the standardized message body to business queue for downstream service consumption and to Data-Flow queue for Flink stream processing engine layer processing through dedicated thread pool; Data-Flow queue event triggers Flink engine to dynamically build DSL configured JobGraph pipeline, and complex event detection and rolling window aggregation operation are performed synchronously during execution, wherein, window aggregation operation calls RocksDB state backend to store results and activates TTL, while real-time monitoring of load indicators dynamically adjusts task manager resource allocation; Periodically perform Checkpoint snapshot persistence operator state during stream processing, and if back pressure signal is detected, reverse control Kafka source end intake rate; Integrate CEP engine detection and window aggregation result input rule engine, statistical model and machine learning detector, calculate abnormal score after associating context library, trigger real-time blocking, manual review or risk marking according to hierarchical threshold, and feedback the disposal result to rule library and model optimization module to realize closed-loop learning; Through unified event model and pluggable connector, realize multi-source data access, adopt zero-intrusion data-driven technology to automatically capture changes; through double-channel message queue and full-link tracking technology, guarantee asynchronous decoupling, end-to-end tracking and Exactly-Once semantics of data processing; and based on Flink state management and multi-level anomaly detection system, solve the problems of state expansion and abnormal early warning.

Images