Intelligent data stream processing system based on Flink and implementation method thereof

Abstract

The invention discloses an intelligent data stream processing system based on Flink and an implementation method thereof, and belongs to the technical field of distributed streaming data processing, and the implementation method comprises the following steps: a data source access layer uniformly accesses multi-source data; the entity monitoring layer captures service entity change in a non-intrusive manner based on JPA, generates a standardized message body and asynchronously delivers the standardized message body to double channels; the unified event model layer maps the data into a standardized event and expands the standardized event; the Flink stream processing engine layer is used for complex event processing, state management and window calculation, and supports declarative pipeline definition and dynamic construction; the anomaly detection layer performs multi-dimensional anomaly detection; the distributed task scheduling layer schedules tasks according to a DAG model, and realizes load balancing through hotspot splitting and state transition; and the result storage and visualization layer adopts a cold and hot separation strategy to store data, and provides a dynamic visualization interface. The real-time performance, the consistency and the intelligent level of data processing are remarkably improved, and the development and maintenance cost is reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of distributed stream data processing, and particularly relates to an intelligent data stream processing system based on Flink and an implementation method thereof. BACKGROUND

[0002] With the deepening of enterprise informatization and digital transformation, the amount of data generated by business systems is growing explosively. Traditional enterprise-level data processing systems mainly adopt batch processing mode, which cannot meet the needs of real-time data processing. Although stream processing technology has developed in recent years, the existing enterprise-level data stream processing system still has the following technical problems in practical application:

[0003] Multi-source heterogeneous data access problem: The data formats and structures of multiple business systems within an enterprise are different, making data integration difficult. Existing systems need to develop special access solutions for each data source, which is high in development and maintenance cost.

[0004] Real-time and consistency guarantee problem: Traditional data processing mainly adopts batch processing or timing polling mechanism, which is difficult to meet the real-time requirements; simple stream processing cannot guarantee data consistency in distributed environment, especially in system failure scenarios.

[0005] Business entity change listening problem: The state change of a large number of business entities in an enterprise system needs to be captured and processed in a timely manner. Existing systems often use intrusive code implementation, and lack a unified change listening and processing mechanism.

[0006] Insufficient intelligence: Existing systems mainly focus on data transmission and simple processing, lack intelligent data analysis and anomaly detection capabilities, and cannot effectively support the data processing needs of complex business scenarios.

[0007] Lack of scalability and fault tolerance: With the growth of data volume, system performance can easily become a bottleneck; the fault recovery mechanism in distributed environment is not perfect, affecting system reliability.

[0008] High development complexity: Developers need to focus on underlying details such as state management and fault recovery, increasing development difficulty and maintenance cost.

[0009] In the prior art, stream processing frameworks such as Apache Storm and Apache Spark Streaming provide certain stream processing capabilities, but have limitations in state management, exactly-once semantic guarantee, and complex event processing, which cannot meet the comprehensive needs of enterprise-level applications.

[0010] Therefore, an intelligent data stream processing system based on an advanced stream processing framework is extremely needed, which can realize unified access of multi-source heterogeneous data, real-time intelligent processing, distributed high-availability deployment, reduce development complexity, and improve maintainability and scalability of the system. SUMMARY

[0011] To solve the above technical problems, the application provides an intelligent data stream processing system based on Flink and an implementation method thereof to solve the problems of the prior art.

[0012] To achieve the above object, the application provides an intelligent data stream processing system based on Flink.

[0013] The data source access layer is configured with a pluggable connector for connecting multiple types of data sources to realize unified access and standardized conversion of data.

[0014] The entity monitoring layer automatically captures business entity change events and generates standardized message bodies based on the JPA entity monitoring mechanism, and asynchronously delivers them to the double channel; wherein JPA is the Java Persistence Application Interface.

[0015] The unified event model layer is used to generate standardized events and extend them after routing to the Flink stream processing engine layer according to the standardized message bodies mapped by the data source access layer.

[0016] The Flink stream processing engine layer is used for complex event processing, state management, window calculation, stream table conversion, and declarative pipeline definition and dynamic construction, and Flink is the Apache Flink stream processing framework.

[0017] The anomaly detection layer is used for anomaly detection of data streams based on rule engines, statistical models and machine learning algorithms.

[0018] The distributed task scheduling layer schedules tasks based on the DAG dependency model, and realizes load balancing through hot spot splitting and state migration; wherein DAG is a directed acyclic graph.

[0019] The result storage and visualization layer is used to store data using a hot and cold separation strategy, and provides a dynamic visualization interface.

[0020] The application also provides an implementation method of an intelligent data stream processing system based on Flink, which comprises the following steps:

[0021] The business entity change is intercepted through the life cycle annotation of the JPA entity listener, a recursive delayed loading algorithm is triggered to generate a pre-change snapshot and a post-change snapshot, then after filtering sensitive fields by analyzing the @FieldMetadata annotation, a unique identifier of the business entity change event is bound by splicing the entity primary key, class name and SnowFlake algorithm, and a standardized message body is obtained, wherein the @FieldMetadata is a field metadata annotation;

[0022] The standardized message body is pushed to the business queue for downstream service consumption and to the Data-Flow queue for Flink stream processing engine layer processing through a dedicated thread pool;

[0023] The Data-Flow queue event triggers the Flink engine to dynamically build a DSL configured JobGraph pipeline, and during the execution process, CEP complex event detection and rolling window aggregation operations are synchronously performed, wherein the window aggregation operation calls a RocksDB state backend to store the result and activates a TTL cleaning mechanism, and at the same time, load indicators are monitored in real time to dynamically adjust TaskManager resource allocation, and the TaskManager is the smallest execution unit of the Flink job;

[0024] During the stream processing process, the Checkpoint snapshot persistence operator state is periodically executed, and if a back pressure signal is detected, the Kafka source end intake rate is reversely regulated and controlled;

[0025] The CEP engine detection and window aggregation result are input into a rule engine, a statistical model and a machine learning detector, an abnormal score is calculated according to the associated context library, and real-time blocking, manual review or risk marking are triggered according to the hierarchical threshold, and the disposal result is fed back to the rule library and the model optimization module to realize closed-loop learning.

[0026] Compared with the prior art, the present application has the following advantages and technical effects:

[0027] The present application designs a flexible and extensible event model and connector architecture to realize unified access of multi-source heterogeneous data; based on the JPA entity life cycle listening mechanism, without modifying the business code, entity change events are automatically captured, and the system coupling degree is significantly reduced; the innovative design of the double-channel architecture and the end-to-end TraceId tracking solves the data traceability and idempotent processing problems in the distributed environment; the rule engine, statistical model and machine learning algorithm are integrated to form a complete anomaly detection system, and the intelligent level of data analysis is significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0028] The drawings constituting a part of this application are used to provide further understanding of the application, the illustrative embodiments of the application and the description thereof are used to explain the application, and do not constitute improper limitation on the application. In the drawings:

[0029] Figure 1 Figure 1 is a schematic diagram of the overall architecture of the system according to an embodiment of the present application;

[0030] Figure 2 Figure 2 is a schematic diagram of the data flow processing flow according to an embodiment of the present application;

[0031] Figure 3 Figure 3 is a schematic diagram of the distributed task scheduling mechanism according to an embodiment of the present application;

[0032] Figure 4 Figure 4 is a schematic diagram of the state management mechanism according to an embodiment of the present application;

[0033] Figure 5 Figure 5 is a schematic diagram of the exception detection flow according to an embodiment of the present application;

[0034] Figure 6 Figure 6 is a schematic diagram of the business document flow interface according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] It should be noted that the embodiments and features in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0036] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0037] Embodiment One

[0038] As shown in Figure 1 , the present embodiment provides an intelligent data flow processing system based on Flink, which adopts hierarchical design, and the specific design of each level is as follows:

[0039] (I) Data source access layer, responsible for connecting various data sources, realizing data collection and preprocessing; the design is as follows:

[0040] The data source access layer adopts plug-in design, supporting unified access of multiple data sources. The main components include:

[0041] Data source connector (SourceConnector): provides a standardized data source connection interface, supporting various data sources such as relational databases, message queues, and file systems;

[0042] Data transformer (DataTransformer): converts raw data into a standard internal format;

[0043] MetadataManager: Manage metadata information of data source, support dynamic configuration and management of data source.

[0044] The core interface design of data source connector adopts standardized interface definition, including three core methods of initializing connector, creating Flink data source and closing connector, to realize unified abstraction of different types of data sources.

[0045] (2) Entity listening layer The entity listening layer is based on the JPA entity listening mechanism, which automatically captures the change events of business entities, converts them into standardized messages and pushes them to the message queue, realizing non-intrusive data-driven;

[0046] The core implementation of entity listening is the DataDrivenEntityListener class, which captures the add, update and delete operations of entities through JPA entity listening annotations (@PostPersist, @PostUpdate, @PostRemove), and then sends the change events to the message queue.

[0047] The entity listening layer adopts a non-intrusive data-driven design, and its core technical ideas and key technical means are as follows:

[0048] (1) Non-intrusive event capture mechanism;

[0049] Through the lifecycle callback annotations @PostPersist, @PostUpdate and @PostRemove provided by JPA, all persistent entities are uniformly annotated with listening logic, without the need to explicitly write event sending statements in business code;

[0050] This mechanism uses the proxy object interception function of JPA to perceive entity state changes before the transaction is successfully committed, ensuring the real-time and consistency of "add, update and delete" events.

[0051] (2) Consistency algorithm of pre-change and post-change snapshots;

[0052] The listening logic first queries the "pre-change entity" as the old snapshot (OldSnapshot) according to the primary key in the read-only transaction, and holds the "new snapshot" (New Snapshot) after the current persistence operation; At the same time, in order to ensure the integrity of the snapshot field, a recursive delayed loading initialization algorithm (recursiveInitliaze) is proposed, which is described as follows:

[0053] Use Hibernate API to determine whether the object or collection is initialized;

[0054] Load all lazy loading fields in depth-first recursion, and use Set <object>De-duplication avoids circular references.

[0055] Limit the maximum recursion depth by maxCompareDepth, balance integrity and performance.

[0056] The final data snapshot meets the three characteristics of "consistent + complete + no circular dependency", laying the foundation for subsequent difference calculation and audit traceability.

[0057] (3) Field difference filtering strategy;

[0058] Introduce self-developed annotations @FieldMetadata and tool class FieldMetadataResolver to mark "no need to flow" fields (such as large fields, static configuration fields, etc.) at the entity class metadata level;

[0059] The listener uses reflection to convert the fields to be excluded into path expressions (for example, src.attachments, dist.attachments[*].content), and writes them into the excludeFields set;

[0060] Implement accurate exclusion of serialization levels through two-dimensional prefix filtering, which not only ensures message slimming, but also avoids leaking sensitive fields.

[0061] (4) Global unique TraceId generation algorithm;

[0062] Form a 64-bit globally unique TraceId by combining the Twitter SnowFlake algorithm and the application instance ID. Specifically, TraceId= <entity primary key>+"_"+<entity class>+"_"+ <snowflakeid>.

[0063] TraceId is used in MQ-Topic, Flink state storage, and audit logs to realize end-to-end link tracking through a unified identifier transmission mechanism.

[0064] The TraceId generated by combining the entity primary key and SnowFlakeId is used as a unique identifier for the entire link, and a global ID matching mechanism is used to support end-to-end tracking and idempotent control.

[0065] (5) Asynchronous high-throughput message delivery mechanism;

[0066] The listener submits asynchronous tasks through the "datadriven" dedicated thread pool provided by the ThreadPoolExecutorFactoryBean;

[0067] This mechanism uses a dual-channel delivery strategy:

[0068] 1. aSyncSendMQ→business queue (used for decoupling downstream microservices);

[0069] 2. aSyncDataFlowMQ→Flink Data-Flow queue (entering the real-time stream computing link);

[0070] The thread pool parameters (core thread count, queue size) are dynamically adjusted by YAML configuration according to business TPS to ensure smooth writing to MQ during extreme peak periods and avoid blocking business transactions.

[0071] (6) The role and association of the Exactly-Once semantic guarantee in the entire link of "data change→event capture→standardized message→real-time stream processing". Through this mechanism, the system can maintain data consistency in high-concurrency scenarios, even in simulated network failure and node crash scenarios, with a data loss rate of less than 0.001%, improving data accuracy by 99.9% compared to traditional At-Least-Once semantic message processing mechanisms.

[0072] In the "standardized message→real-time stream processing" link, the MQ end adopts the "transaction message + idempotent consumption" mechanism to ensure that messages are not lost or duplicated during transmission; combined with the Flink Checkpoint snapshot replay mechanism, data processing consistency is guaranteed in the stream processing stage.

[0073] Logical association with other levels of design:

[0074] Association with the entity listening layer: the entity listening logic and the main business transaction belong to the same database transaction domain, ensuring that messages are sent only after the transaction is successfully committed;

[0075] Associated with the unified event model layer: message deduplication and idempotent processing are realized through the TraceId mechanism;

[0076] Associated with the Flink stream processing engine layer: state consistency is guaranteed by using the Checkpoint mechanism;

[0077] Associated with the result storage layer: consistency of processing results is ensured when writing.

[0078] Through the exactly-once semantics guarantee, it is ensured that in a distributed environment, even if abnormal conditions such as network failure and node crash occur, data can be processed exactly once; at the same time, data loss and repeated processing are prevented, and the accuracy and consistency of business data are guaranteed.

[0079] (7) Extensible and pluggable framework design;

[0080] In the "event capture → standardized message" link, the variable part (MQ sender, field parser, etc.) in the listening logic is injected in the form of a Spring Bean, supporting replacement and extension; the interface DataDrivenSender allows access to multiple message middleware, improving the cross-platform adaptation capability of the system.

[0081] Logical association with other levels of design:

[0082] Associated with the data source access layer: through standardized interface definition, unified access to multiple data sources is supported;

[0083] Associated with the unified event model layer: through the extensible event model and Schema management, dynamic addition of new event types is supported;

[0084] Associated with the Flink stream processing engine layer: through DSL configuration and dynamic pipeline construction, flexible definition of processing logic is supported;

[0085] Associated with the anomaly detection layer: plug-in integration of multiple detection algorithms is supported.

[0086] Through extensible and pluggable framework design, the coupling between components of the system is reduced, the maintainability and scalability of the system are improved, the rapid adaptation of business scenarios and the smooth upgrade of technical components are supported, and an architectural foundation is provided for long-term evolution and technical iteration of the system.

[0087] Through the above technical solutions, the present application realizes the automatic connection of the whole link "data change → event capture → standardized message → real-time stream processing" without modifying the business service code, significantly improving the real-time performance, maintainability and intelligent level of enterprise-level applications.

[0088] The business entity class needs to add the @EntityListeners annotation to specify the DataDrivenEntityListener as the entity listener. When the business entity changes, the DataDrivenEntityListener will capture the change event, get the state before and after the entity changes, build a standardized message body (DataDrivenMqBody), and then send it to the message queue asynchronously. The standardized message body contains traceId (unique identifier of the change event), queue (name of the message queue), src (entity object before change), and dist (entity object after change) information.

[0089] (Three) Unified event model layer design;

[0090] The unified event model layer is a bridge connecting the entity listener layer and the Flink stream processing engine layer. It converts data of different sources and formats into a unified event model for subsequent processing. The core of the unified event model is the BusinessEvent class, which contains basic information of the event (such as event ID, event type, event time, event source, etc.) and specific business data. The technical features and implementation methods of the unified event model include:

[0091] (1) Event standardization design;

[0092] The system designs a two-layer event structure containing metadata and business data, with the following specific design:

[0093] "BusinessEvent{

[0094] / / Metadata part

[0095] String eventId; / / Event unique identifier

[0096] String eventType; / / Event type

[0097] long eventTime; / / Event timestamp

[0098] String eventSource; / / Event source

[0099] / / Business data part

[0100] Object businessData; / / Business data object

[0101] }”。

[0102] The event type adopts a hierarchical namespace design: {domain}.{subdomain}.{action}, such as order.purchase.created, to ensure global uniqueness; the eventId is automatically consistent with the TraceId mentioned above, realizing the association between event tracking and entity changes.

[0103] (2) Schema dynamic management mechanism;

[0104] The dual-mode support strategy of JSON Schema + Avro Schema is adopted, in which JSON Schema is used for flexible processing of variable business data, and Avro Schema is used for high-performance serialization / deserialization;

[0105] By designing the SchemaRegistry component, the Schema definitions of all event types are managed centrally in ZooKeeper.

[0106] The version number marking and compatibility checking mechanism is used to support Schema evolution, and the forward and backward compatibility is realized through field default values and compatibility rules.

[0107] (3) Event conversion and routing algorithm;

[0108] Event conversion: by defining a general converter interface EventConverter<S, T> and providing multiple implementation classes, data from different sources is converted into unified BusinessEvent.

[0109] Routing strategy: use the matching rules based on event type and event metadata to determine which Flink processing pipeline the message is routed to.

[0110] Filter enhancement: with the help of the responsibility chain mode of Filter Chain and Enrichment Chain, the event is filtered and enhanced.

[0111] Supplementary explanation of the event enhancement step: first, the event converter converts the standardized message body into an initial BusinessEvent object; then, the event enhancer receives this initial event object; the event enhancer obtains additional context information by querying external data sources (such as databases, caches, APIs, etc.); the obtained context information is added to the attributes of the event object to form an enhanced event; finally, the enhanced event is passed to the event router for subsequent processing.

[0112] The event enhancer adopts the chain of responsibility pattern to realize multi-level enhancement processing; through the configurable enhancer chain, flexible enhancement strategies are supported; each enhancer focuses on adding specific types of context information (such as dimension data, associated entities, historical data, etc.).

[0113] (Four) Flink stream processing engine layer design;

[0114] The Flink stream processing engine layer is the core processing component of the system, which realizes powerful stream processing capabilities based on Apache Flink, supports complex event processing, state management, window calculation, stream table conversion, etc. The technical characteristics and implementation methods of the Flink stream processing engine layer are as follows:

[0115] (1) Dynamic pipeline construction framework;

[0116] A declarative pipeline definition DSL is designed to describe the stream processing flow in YAML or JSON format; based on the DSL configuration, the system calls the FlowBuilder interface to parse the configuration content, dynamically constructs and deploys the Flink JobGraph; the component-based design pattern is used to realize the hot plug function of the pipeline components, so that a single processing component can be dynamically updated without stopping the entire process.

[0117] (2) Distributed state management and fault tolerance mechanism;

[0118] The state backend of Flink is used for efficient state management, supporting three state storage methods: memory, file system, and RocksDB.

[0119] A double-layer state management strategy is designed, hot data is saved in the Flink state for real-time processing, and cold data is stored in an external database for historical query. The state manager is an interface component between the Flink stream processing engine layer and the state management layer, responsible for calling the state storage, retrieval, and management functions provided by the state management layer.

[0120] By integrating the Flink Checkpoint mechanism and extending the Savepoint function, a double-layer fault tolerance system is built, which automatically recovers the processing state using the latest checkpoint data when a system failure occurs, ensuring data consistency during fault recovery.

[0121] By configuring the TTL (Time-To-Live) parameter of the Flink state backend and combining a custom cleanup strategy, long-time unaccessed state data is automatically marked and cleaned up, avoiding the problem of state expansion and improving the stability of long-term system operation.

[0122] (3) Complex event processing engine (CEP Engine);

[0123] Complex event processing engine is implemented based on Flink CEP, which supports event pattern definition and detection; a visual pattern editor is designed, which supports business personnel to define complex event patterns through drag-and-drop; multi-dimensional event correlation algorithms are implemented, such as TraceId correlation, time window correlation, entity ID correlation, etc.; both positive patterns (such as event sequence) and negative patterns (such as timeout detection) are supported.

[0124] (4) Adaptive flow control and back pressure mechanism;

[0125] By monitoring the back pressure (back pressure) indicators of Flink task, combined with the threshold control algorithm, when detecting the back pressure of downstream processing components, the intake rate of upstream data source is automatically adjusted to realize the dynamic balance of system load and guarantee the processing stability; multi-level cache and batch processing optimization are realized to balance throughput and delay; configurable data sampling and aggregation strategies are provided to guarantee system stability in high traffic scenarios.

[0126] The Flink stream processing engine layer can be divided into pipeline building unit, event pattern detection unit, state calculation unit, data enhancement unit and fault tolerance unit;

[0127] The pipeline building unit is used to parse the DSL configuration, trigger the FlowBuilder to dynamically build the JobGraph, the DSL configuration is the configuration of the Flink job, the FlowBuilder is the stream builder interface, and the JobGraph is the job topology graph;

[0128] The event pattern detection unit is used to scan the event stream through the CEP engine, match the preset complex pattern, trigger the alarm or aggregation operation, wherein the CEP engine is a complex event processing engine;

[0129] The state calculation unit is used to execute window aggregation, call the RocksDB state backend to store the intermediate results, and synchronize to activate TTL to clean up expired states; wherein TTL is a survival time mechanism; RocksDB is an embedded key-value storage database;

[0130] The data enhancement unit is used to associate external dimension tables to expand event attributes;

[0131] The fault-tolerant unit ensures data consistency during fault recovery through a checkpoint and savepoint-based fault-tolerant mechanism. Both checkpoint and savepoint are mechanisms in the Flink stream processing framework. Checkpoint is a mechanism in Flink that automatically saves the state of a stream processing job at regular intervals, while savepoint is a mechanism in Flink that manually triggers the saving of the state of a stream processing job.

[0132] The state management layer manages state information during stream processing to ensure data consistency during fault recovery. State management is one of the core capabilities of the Flink stream processing engine layer, and the state manager invokes the functions contained in this layer. The technical features and implementation methods of this layer are as follows:

[0133] (1) Multi-level state storage architecture:

[0134] A multi-level state storage architecture is designed, which includes memory, RocksDB, and external storage. The appropriate storage level is selected according to the data access frequency and importance. Hot state data is stored in memory to support high-frequency and low-latency access. Large-scale state data is stored in RocksDB to balance performance and capacity. Historical state data is periodically archived to external storage systems for historical query and analysis.

[0135] (2) State life cycle management:

[0136] The state TTL (Time-To-Live) mechanism is innovatively introduced to set a reasonable survival period for each type of state data. A dual-TTL policy based on access time and update time is implemented to more accurately control the life cycle of state data. A state cleanup scheduler is designed to periodically trigger the cleanup of expired state data, avoiding the growth of state data and causing memory pressure and performance degradation. Incremental cleanup of state data is supported to avoid performance impact on real-time processing during the cleanup process.

[0137] (3) Distributed state consistency guarantee:

[0138] By integrating the checkpoint mechanism of Flink and extending its trigger conditions, distributed state consistency snapshots are realized to support state reconstruction during fault recovery. Adaptive trigger frequency and hierarchical storage methods are used to optimize the trigger strategy and storage method of checkpoint, balancing consistency guarantee and performance overhead. The incremental checkpoint mechanism is implemented using the CDC (Change Data Capture) technology to persist only the changed state data, reducing the overhead of checkpoint. The state version management mechanism is designed using timestamp and version number dual marking to support state rollback and historical state query.

[0139] (4) State query and access optimization:

[0140] An efficient state indexing mechanism is implemented by combining B+ tree and inverted index, supporting multi-dimensional queries of state data by primary key, time range, attribute value, etc. A multi-level cache strategy is used to design the state cache layer, which caches frequently accessed state data to reduce query latency. State asynchronous access API is provided based on Future / Promise mode to avoid blocking the main processing flow. Vectorized batch processing technology is used to support batch read and write of state data, improving throughput.

[0141] (5) State migration and load balancing:

[0142] Key Group-based state partitioning strategy is implemented to support uniform distribution of state data. A state hotspot detection mechanism is designed to identify uneven state access. Online migration of state data is supported to achieve dynamic load rebalancing. Data transmission and synchronization strategies during state migration are optimized to minimize the impact of migration on real-time processing.

[0143] The state management module is tightly integrated with the Flink stream processing engine, providing strong state support for complex event processing, window calculation, pattern detection, etc. Through innovative state life cycle management and distributed consistency guarantee mechanism, the problem of state expansion caused by long-time running is effectively solved, improving the stability and reliability of the system.

[0144] Figure 4 The hierarchical design and data flow of state management are demonstrated, including key state (value state, list state and map state), multiple state backend implementations (in-memory state backend, file system state backend and RocksDB state backend), checkpoint mechanism, state recovery process and generation and conversion of computing state.

[0145] (Six) Abnormality detection layer design;

[0146] The abnormality detection layer is based on multiple detection strategies and algorithms to identify and handle abnormal patterns in data streams. The abnormality detection layer integrates rule engines, statistical analysis and machine learning models to form a multi-level and comprehensive abnormality detection system. The technical features and implementation methods of abnormality detection are as follows:

[0147] (1) Multi-dimensional abnormality detection strategy;

[0148] Rule Engine Detection: Based on pre-defined business rules for anomaly detection, the rule execution engine is implemented by integrating Drools rule language and execution engine API, supporting complex condition expressions and rule chains; the rule file dynamic loader and version control mechanism are used to support dynamic loading and hot updating of rules without restarting stream processing jobs.

[0149] Statistical Model Detection: Based on statistical methods for data modeling and anomaly detection, multiple statistical methods including Z-Score, MAD (Median Absolute Deviation), EWMA (Exponential Weighted Moving Average) are implemented through sliding window calculation and probability distribution fitting techniques; adaptive threshold adjustment is supported by historical data analysis and seasonal adjustment algorithms, dynamically setting anomaly judgment criteria based on data distribution characteristics.

[0150] Machine Learning Detection: Utilizing machine learning algorithms for complex pattern recognition and anomaly detection, multiple anomaly detection algorithms are supported through feature engineering and model encapsulation techniques; online learning and offline learning are combined through incremental learning and batch training, and model evaluation and automatic parameter tuning mechanisms are used to support continuous model optimization.

[0151] (2) Context-aware anomaly evaluation mechanism;

[0152] Anomaly Context Repository is designed to store context information such as time, environment, and related events of anomaly occurrence; a context-based anomaly scoring algorithm is implemented to consider multiple detector results and historical information; a business weight matrix is introduced to assign different weights to different types of anomalies based on business importance.

[0153] The specific allocation rules and business importance acquisition mechanism of the business weight matrix are as follows:

[0154] 1) Business importance acquisition method:

[0155] The system obtains the basic business weight from the business domain configuration file maintained by the configuration center; supports dynamic adjustment of weight values by business experts through the management interface according to actual business conditions; combines historical anomaly impact analysis data to automatically optimize weight configuration.

[0156] 2) Weight allocation rules:

[0157] The weight relationship is stored in a two-dimensional matrix structure of business domain and exception type; the weight value of core business processes (such as payment transaction and order confirmation) is set to be in the range of 0.8-1.0, and the weight value of non-core business processes (such as log recording and statistical analysis) is set to be in the range of 0.1-0.5; the system dynamically adjusts the maximum weight according to time factors (such as weight coefficient of 1.5 in business peak period and 1.0 in non-peak period) and context information (such as exception influence range coefficient: 1.5 for global influence and 1.0 for local influence);

[0158] The weight calculation formula is: maximum weight = basic weight × time factor × influence factor;

[0159] This business semantics-based weight mechanism ensures the consistency of the exception detection result with the actual business priority, effectively improving the pertinence and value of exception handling.

[0160] (3) Self-learning exception pattern library;

[0161] An exception pattern library (Pattern Repository) is designed to store the discovered exception patterns and their characteristics; a pattern learning mechanism based on user feedback is realized through user feedback collection and weight adjustment algorithm to continuously optimize the exception detection effect; a similar pattern search is supported by using the vector space model and cosine similarity calculation, and new exceptions similar to historical exceptions are quickly identified through similarity calculation.

[0162] (4) Exception early warning and intervention mechanism;

[0163] A multi-level early warning mechanism is realized through exception scoring and notification channel selection strategy, and different levels of alarms are issued according to the severity of the exception; a preventive intervention strategy is designed by using trend prediction and threshold warning technology to provide risk warning and preventive measure guidance before the occurrence of the exception; a rule-driven automated operation and manual confirmation process are combined to support a processing flow combining automated response and manual intervention.

[0164] The exception detection layer includes a multi-dimensional parallel detection unit, a context-aware assessment unit, and a warning and response unit.

[0165] The multi-dimensional parallel detection unit is used to obtain rule matching results, index deviation scores, and abnormal probability scores and input them to the context-aware assessment unit.

[0166] The context-aware assessment unit is used to obtain an exception score with context labels based on rule matching results, index deviation scores, and abnormal probability scores.

[0167] The warning and response unit performs hierarchical warning and response based on the exception score with context labels, and feeds back the disposal result to the exception pattern library.

[0168] As shown in Figure 5 The anomaly detection process of the embodiment of the present application adopts a multi-level detection and processing architecture. The data stream is first subjected to feature extraction, and then divided into three parallel processing paths: a pattern recognition path (including statistical anomaly detection and machine learning models), a rule calculation path (processed by a rule engine). The results of the three paths are converged to a threshold judgment link, which comprehensively evaluates the abnormal situation, and is uniformly processed by an alarm management module. The anomaly processing is divided into two categories: notification and automatic response. The notification includes email notification, SMS notification and Webhook callback; the automatic response includes automatic retry, resource adjustment and fault recovery. Figure 5 The complete process from data input to anomaly response is clearly shown, and the actual operation mechanism of the above-mentioned anomaly detection technology is intuitively presented, especially the key links of multi-dimensional parallel detection and hierarchical response, which ensures the accurate identification and efficient processing of the system for various abnormal situations.

[0169] (Seven) Design of distributed task scheduling layer;

[0170] The distributed task scheduling layer is responsible for intelligent scheduling and resource management of stream processing tasks, ensuring high availability, scalability and resource utilization efficiency of the system. Based on the distributed architecture of Flink, the layer realizes intelligent scheduling of tasks and dynamic allocation of resources. The technical features and implementation methods of distributed task scheduling are as follows:

[0171] (1) Intelligent resource allocation algorithm;

[0172] Through monitoring index collection and load prediction model, a load-aware resource allocation algorithm is realized, which dynamically adjusts resource allocation according to task complexity, data volume and processing time; using quota limit and priority strategy, a resource quota management mechanism is designed to reasonably allocate computing resources and avoid resource contention; using automatic scaling trigger and resource scheduler, elastic scaling is supported, and task parallelism and resource configuration are automatically adjusted according to traffic changes.

[0173] (2) High-availability cluster management;

[0174] Through the temporary node and listener mechanism of ZooKeeper, a distributed coordination and leader election mechanism is realized to ensure the high availability of cluster management; using heartbeat detection, timeout judgment and fault isolation strategy, a multi-level fault detection and recovery strategy is designed, including process-level, node-level and cluster-level fault handling; using version control and traffic switching technology, rolling upgrade and blue-green deployment support are realized to ensure smooth upgrade and migration of the system.

[0175] (3) Task dependency management and scheduling optimization;

[0176] A DAG (Directed Acyclic Graph)-based task dependency model is designed to accurately describe the dependency relationship between tasks; a topological sorting and critical path analysis algorithm is implemented to optimize the task execution order; and a dynamic scheduling based on data readiness is supported to improve resource utilization efficiency.

[0177] (4) Load balancing and hot spot migration;

[0178] A dynamic rebalancing algorithm for data partition is implemented to solve the data skew problem; a hot task detection and splitting mechanism is designed to dynamically decompose high-load tasks into multiple subtasks; and task and state migration is supported to optimize overall resource utilization.

[0179] The distributed task scheduling layer can be divided into a resource perception unit, an intelligent decision unit, an elastic execution unit and a fault self-healing unit.

[0180] The resource perception unit is used to report the CPU, memory, back pressure and data skew of each node in real time through the monitor, and aggregate the load portrait through the resource manager;

[0181] The intelligent decision unit is used to call the load perception algorithm based on the load portrait through the task scheduler to calculate the optimal parallelism and resource quota that meet the DAG dependency model, and if a hot task is found, the splitting is triggered and the subtask topology is generated;

[0182] The elastic execution unit is used to issue the decision of the intelligent decision unit through the ZooKeeper cluster manager, to adjust the resource allocation and migrate the state online, to complete the version rolling update, and the ZooKeeper is a distributed coordination service;

[0183] The fault self-healing unit is used to activate the standby node on standby in case of process, node or cluster level exception through the fault recovery manager, to recover the job state based on the latest Savepoint, and to reschedule the task immediately.

[0184] As shown in Figure 3 , the distributed task scheduling mechanism of the embodiment of the application adopts a hierarchical task management architecture to achieve efficient task allocation and resource scheduling. The mechanism is composed of a resource manager, a job manager, a checkpoint coordinator and a task manager, realizes intelligent scheduling, dynamic load balancing, fault self-healing and elastic scaling based on the DAG dependency model, and effectively solves the data skew and hot task problems.

[0185] (8) Result storage and visualization layer design;

[0186] The result storage and visualization layer is the presentation layer of the system, responsible for handling the storage, querying, and visualization of results, providing users with intuitive data insights and decision support. This layer employs a multi-level storage architecture and flexible visualization design to achieve efficient data management and display. The technical features and implementation methods of result storage and visualization are as follows:

[0187] (1) Multi-level storage architecture;

[0188] An efficient real-time query processing engine is implemented through query parsing and execution framework, supporting SQL and SQL-like query languages; a query optimizer is designed using execution plan optimization techniques to automatically select the best execution plan; multi-dimensional analysis and real-time aggregation are supported based on dimension index methods, such as time dimension, spatial dimension, entity dimension, etc.; query result caching is achieved through a cache management mechanism to improve query performance.

[0189] The multi-level state storage architecture of the state management layer has the following essential differences:

[0190] Data property difference: the result storage layer handles the final business result data after stream processing, facing the final user query and analysis; while the state management layer handles the intermediate state data during stream computing, mainly used for computing recovery and stream processing logic.

[0191] Access mode difference: the data of the result storage layer mainly faces the query access of external applications and users, requiring support for complex query modes and high concurrency reading; the data of the state management layer mainly faces the computing process of Flink, with efficient state update and checkpoint generation as the main goal.

[0192] Storage cycle difference: the data of the result storage layer is usually saved for a longer period of time, requiring complete data lifecycle management; the data of the state management layer usually only exists during the running of the stream processing job, and may not be needed after the job ends.

[0193] Technical implementation difference: the result storage layer uses Redis, ClickHouse, Elasticsearch, HDFS, and other application-oriented storage systems; the state management layer mainly uses Flink's built-in state backends such as memory, RocksDB, and other storage optimized for stream processing.

[0194] The multi-level storage architecture of the layer is specifically implemented as follows: hot data is stored in a memory database or a distributed cache, supporting high-frequency access; warm data is stored in a columnar storage system, supporting fast aggregation query; cold data is stored in a distributed file system, supporting large-scale historical data archiving; a data life cycle management mechanism is designed to automatically complete the hot / cold migration of data; and multiple storage backends such as Redis, ClickHouse, Elasticsearch, HDFS and the like are supported to meet different scene requirements.

[0195] (2) Real-time query engine;

[0196] An efficient real-time query processing engine is implemented to support SQL and SQL-like query languages; a query optimizer is designed to automatically select the best execution plan; multi-dimensional analysis and real-time aggregation such as time dimension, space dimension, entity dimension and the like are supported; and a query result caching mechanism is implemented to improve query performance.

[0197] (3) Dynamic visualization framework;

[0198] A componentized visualization framework is designed through component registration and layout management system to support multiple visualization modes such as charts, dashboards, heat maps and the like; data-driven visualization components are implemented by using data feature recognition and display rule matching to automatically select appropriate display forms according to data characteristics; an interactive analysis is supported by using event listening and state management mechanism, such as drilling down, filtering, comparison and the like; a visualization template library is provided by using template definition language and rendering engine to facilitate quick construction of business dashboards.

[0199] (4) Real-time alarm and notification system;

[0200] A multi-channel alarm pushing mechanism is designed by using message queues and notification adapters to support email, short message, enterprise instant messaging tools and the like; alarm intelligent grouping and merging algorithms are implemented by using similarity calculation and time window aggregation to avoid alarm storm; and alarm escalation mechanism is supported by using time threshold and escalation rule engine to automatically escalate according to alarm duration and severity.

[0201] The result storage and visualization layer can be divided into a result writing unit, a life cycle migration unit, a real-time query unit, a dynamic visualization unit and an alarm unit;

[0202] The result writing unit processes each output of the Flink stream processing engine layer through a Sink to obtain event heat; a storage manager assigns a tag to the event according to the event heat field, and performs multi-level storage according to the tag; wherein the Sink is a receiver operator;

[0203] A life cycle migration unit is configured to automatically perform data migration by a data life cycle management mechanism when a Redis memory occupancy or a ClickHouse partition age triggers a threshold, and synchronize new partition metadata to a SchemaRegistry after the migration is completed; and send a resource release or application signal to a resource manager of a distributed task scheduling layer through an API gateway, wherein the Redis is a Redis memory database, the ClickHouse is a ClickHouse columnar database, and the SchemaRegistry is a schema registry;

[0204] A real-time query unit is configured to retrieve a query result cache according to a query request, generate an optimal execution plan by using multi-dimensional indexing and partition pruning if a hit is not made, perform parallel scanning and aggregation on multi-level storage data, and return an aggregated result;

[0205] A dynamic visualization unit is configured to automatically match a display form from a visualization template library according to data dimensions and features after receiving the aggregated result returned by the real-time query unit; and write an interactive log generated by user operation to an abnormal pattern library in real time to provide an incremental sample for model retraining of an abnormal detection layer;

[0206] An alarm unit is configured to call an alarm intelligent grouping and merging algorithm according to a comprehensive abnormal score to perform multi-channel hierarchical pushing.

[0207] The core components and responsibilities of the result storage and visualization layer are as follows:

[0208] A storage manager is configured to manage a multi-level storage system, implement a cold and hot separation storage strategy, store hot data in a memory database such as Redis, store warm data in a columnar storage system such as ClickHouse, store cold data in a distributed file system such as HDFS, and design a data life cycle management mechanism to automatically complete cold and hot migration of data.

[0209] A query engine is configured to process various data query requests, support SQL and SQL-like query languages, design a query optimizer to automatically select an optimal execution plan, support multi-dimensional analysis and real-time aggregation, and implement a query result cache mechanism to improve query performance.

[0210] A visualization engine is configured to generate various charts and dashboards, design a componentized visualization framework to support various visualization modes such as charts, dashboards, and heat maps, implement data-driven visualization components to automatically select a suitable display form according to data characteristics, support interactive analysis, and provide a visualization template library.

[0211] An alarm center is configured to manage alarm rules and notification strategies, design a multi-channel alarm pushing mechanism to support email, short message, enterprise instant messaging tools, and the like, implement an alarm intelligent grouping and merging algorithm, and support an alarm escalation mechanism.

[0212] API gateway: provides a unified data access interface across multiple levels of storage, provides a RESTful API interface, and uniformly encapsulates access to the multi-level storage system; implements permission control and access throttling; supports data format conversion and protocol adaptation.

[0213] Result writing unit: handles the writing of stream processing results.

[0214] The output of the Flink stream processing engine layer is processed by the Sink operator to obtain event heat; based on the event heat, the event is labeled, and the multi-level storage distribution is performed through the data writing mechanism in the multi-level storage architecture.

[0215] Lifecycle migration unit: manages the lifecycle of data, automatically migrates data when the Redis memory usage or ClickHouse partition age triggers the threshold; updates the metadata after migration; sends resource release or application signals to the resource manager.

[0216] Real-time query unit: performs efficient data query. Retrieve query result cache; use multi-dimensional index and partition pruning to generate optimal execution plan; perform parallel scanning and aggregation on multi-level storage data.

[0217] Dynamic visualization unit: realizes intelligent data visualization.

[0218] According to the data dimension and characteristics, the display form is automatically matched from the visualization template library; user interaction logs are written back to Pattern Repository in real time, providing incremental samples for the anomaly detection layer.

[0219] The result storage is based on the Sink mechanism of Flink, supporting writing processing results to Elasticsearch, Kafka, database and other target systems. Visualization is based on Web technology, providing rich charts and interactive analysis functions. This system realizes flexible result storage and visualization functions, supporting multiple storage methods and visualization display.

[0220] As shown in Figure 6 , the business document flow interface of the system intuitively shows the complete business process from procurement contract to prefabricated invoice. By converting complex data processing results into intuitive business views, it realizes seamless connection between technology and business, and provides real-time and comprehensive business monitoring capabilities for enterprises.

[0221] The system forms an organic whole through innovative technical linkage mechanisms between each level, realizing full-process collaboration from data capture to processing and analysis. The key linkage mechanisms are as follows:

[0222] (a) Full-link data tracking mechanism based on TraceId;

[0223] The TraceId generated by the entity listening layer (by <Entity Primary Key>+"_"+<Entity Class>+"_"+ <snowflakeid>The TraceId is kept as eventId in the unified event model layer, as a state key in the Flink stream processing, for correlation analysis in the anomaly detection layer, and for data provenance in the result storage layer. This end-to-end identification mechanism solves the problems of data tracking and idempotency in distributed systems.

[0224] (b) Dual-channel decoupling and coordination mechanism;

[0225] The entity monitoring layer innovatively designs a dual-channel architecture with business queues and Data-Flow queues in parallel, the former serving business system decoupling and the latter dedicated to real-time stream processing. The two channels are physically isolated but logically unified, sharing the same message body structure and TraceId, achieving loose coupling and data consistency between business systems and real-time analysis systems.

[0226] (c) Layered state management and data consistency coordination mechanism;

[0227] The entity monitoring layer ensures reliable data delivery through transactional messages, the unified event model layer avoids duplicate messages through idempotent processing, the Flink stream processing engine layer guarantees state consistency through the Checkpoint mechanism, and the result storage layer ensures result consistency through atomic writing. This cross-layer coordination mechanism achieves end-to-end Exactly-Once semantics, ensuring data processing accuracy even in the case of network failure or node crashes.

[0228] (d) Cross-layer resource dynamic allocation coordination mechanism;

[0229] The distributed task scheduling layer monitors resource usage across all layers, achieving cross-layer resource coordination: when detecting back pressure in the Flink engine layer, it can control the message sending rate of the entity monitoring layer; when a hot task is found, task splitting is triggered and state distribution is adjusted; when the result storage layer data volume surges, more storage resources are dynamically allocated. This cross-layer resource coordination mechanism ensures system stability under load fluctuations.

[0230] (e) Metadata-driven dynamic processing link construction mechanism;

[0231] Information sharing and processing logic selection are achieved through the transmission and enhancement of metadata between layers. The entity type and operation type captured by the entity monitoring layer serve as metadata, guiding event conversion and routing in the unified event model layer; event metadata drives the Flink stream processing engine layer to dynamically select processing pipelines; anomaly detection results are attached to processing results as metadata, guiding storage strategy selection in the result storage layer. This metadata-driven mechanism makes the system highly flexible and scalable.

[0232] The system also includes a metadata management module for managing various types of metadata information in the system, a security management module for implementing data access control and security auditing, a monitoring management module for monitoring system running state and performance indicators, and a configuration management module for managing system configuration and parameter settings. The system adopts a micro-service architecture design, and the modules interact with each other through standard interfaces, realizing a loosely coupled system architecture that supports independent deployment and expansion of modules.

[0233] As shown in Figure 2 , the system adopts a distributed deployment architecture. The system includes seven core components: data source, data source connector, data converter, Flink engine, state manager, anomaly detector, and result receiver, forming a complete data processing pipeline.

[0234] (1) JPA entity listener implementation:

[0235] Through the JPA entity listener mechanism, automatic capture and event sending of business entity changes are realized. The specific implementation steps are as follows:

[0236] (1) Define entity listener class (DataDrivenEntityListener):

[0237] Capture entity addition, modification, and deletion events through @PostPersist, @PostUpdate, and @PostRemove annotations; obtain entity state snapshots before and after changes in event handling methods; build standardized message bodies and asynchronously send them to the message queue.

[0238] (2) Entity class integrates listener:

[0239] Add the @EntityListeners(DataDrivenEntityListener.class) annotation to the entity class; the entity class needs to implement the BaseEntity interface to provide unique identification and metadata access capabilities.

[0240] (3) Recursive lazy loading initialization:

[0241] For lazy loading fields of entities, recursive initialization is adopted to ensure the integrity of data snapshots; set the maximum recursion depth to avoid circular references and performance problems; use reflection technology to identify and process different types of fields.

[0242] (4) Asynchronous message sending:

[0243] Use a dedicated thread pool to asynchronously process message sending to avoid blocking business transactions; implement a retry mechanism for failed messages to ensure reliable message delivery; use message deduplication and idempotent processing to avoid the impact of duplicate messages.

[0244] (II) Flink Stream Processing Implementation:

[0245] Based on Apache Flink, powerful stream processing capabilities are implemented as follows:

[0246] (1) Stream Processing Job Configuration:

[0247] Stream processing jobs are defined through YAML or JSON configuration files, including data sources, operators, states, windows, etc. Dynamic loading and updating of configurations are supported, allowing adjustment of processing logic without restarting the job.

[0248] (2) Complex Event Processing:

[0249] Complex event pattern definition and detection are implemented based on the Flink CEP framework. Multiple pattern types such as temporal patterns, cyclic patterns, and conditional patterns are supported. Custom pattern matching algorithms are implemented to improve matching efficiency and accuracy.

[0250] (3) State Management:

[0251] RocksDB is used as the state backend to support large-scale state data storage. State expiration and cleanup strategies are implemented to prevent unlimited state growth. Checkpoint and Savepoint mechanisms are used to ensure state consistency and recoverability.

[0252] (4) Window Calculation:

[0253] Time windows (rolling windows, sliding windows, and session windows) and counting windows are supported. Event time processing based on watermarks is implemented to handle out-of-order and late data. Multi-dimensional window aggregation is supported, such as by time, entity ID, and event type dimensions.

[0254] (5) Dynamic Rule Processing:

[0255] Dynamic loading and execution mechanisms for rules are implemented, supporting real-time rule updates. Rules are described using DSL (Domain Specific Language) for ease of understanding and writing by business personnel. Rule version management and rollback are supported to ensure controllability of rule changes.

[0256] (III) Exception Detection Implementation:

[0257] The system implements a multi-level exception detection mechanism as follows:

[0258] (1) Rule Engine Detection:

[0259] Based on Drools, the rule execution engine is implemented to support complex conditional expressions. The rules use a declarative syntax, making it easy for business experts to write and maintain. Hot deployment and version management of rules are supported to enable dynamic updates of rules.

[0260] (2) Statistical model detection:

[0261] The Z-Score algorithm for sliding windows is implemented to detect abnormal fluctuations in numerical indicators. The EWMA (Exponential Weighted Moving Average) model is used to detect trend anomalies in time series data. Adaptive thresholds are supported to dynamically adjust the anomaly determination criteria based on historical data.

[0262] (3) Machine learning detection:

[0263] The Isolation Forest algorithm is integrated for anomaly detection in high-dimensional data. An anomaly detection model based on autoencoders is implemented, suitable for complex pattern recognition. Online and offline hybrid learning is supported to continuously optimize the model.

[0264] (4) Multi-dimensional correlation analysis:

[0265] Full-link correlation is implemented based on TraceId to identify cross-system abnormal patterns. Graph algorithms are used for correlated event analysis to discover potential causal relationships. Time window-based pattern correlation is supported to identify time-series related anomalies.

[0266] (Four) Implementation effect:

[0267] In the practical application of a large manufacturing enterprise, this system realizes real-time data collection and processing of multiple business systems (including ERP, MES, SCM, etc.) within the enterprise, and builds a real-time business monitoring and analysis platform. The system processes more than 10 million data per day, with an anomaly detection accuracy of over 95%.

[0268] Table 1

[0269]

[0270] Table 1 continued 1

[0271]

[0272] Table 1 continued 2

[0273]

[0274] Table 1 continued 3

[0275]

[0276] Table 1 continued 4

[0277]

[0278] Table 1 continued 5

[0279]

[0280] All English terms and self-named terms used in this invention follow the standard definitions in the following technical specifications and documents, which can be obtained from the corresponding official documents and Table 1 for explicit and unique interpretation.

[0281] (1) Java Language Specification: Oracle official specification, document: https: / / docs.oracle.com / javase / specs / ;

[0282] (2) JPA: Java Persistence API specification, document: https: / / jakarta.ee / specifications / persistence / ;

[0283] (3) Apache Flink: Distributed stream processing framework, document: https: / / flink.apache.org / docs / ;

[0284] (4) Spring Framework: Java application framework, document: https: / / spring.io / projects / spring-framework;

[0285] (5) Hibernate ORM: Object-Relational Mapping framework, document:

[0286] https: / / hibernate.org / orm / documentation / ;

[0287] (6) Apache Kafka: Distributed stream processing platform, document: https: / / kafka.apache.org / documentation / ;

[0288] (7) Redis: In-memory data storage system, document: https: / / redis.io / documentation;

[0289] (8) ClickHouse: Columnar database, document: https: / / clickhouse.com / docs / ;

[0290] (9) Elasticsearch: Distributed search engine, documentation: https: / / www.elastic.co / guide / ;

[0291] (10) Apache ZooKeeper: Distributed coordination service, documentation: https: / / zookeeper.apache.org / doc / ;

[0292] (11) RocksDB: Embedded key-value database, documentation: https: / / rocksdb.org / docs / ;

[0293] (12) Drools: Business rule management system, documentation:

[0294] https: / / www.drools.org / learn / documentation.html;

[0295] (13) Machine learning algorithms: Isolation Forest, One-Class SVM, etc. Refer to sklearn documentation: https: / / scikit-learn.org / stable / ;

[0296] (14) Statistical methods: Z-Score, MAD, EWMA, etc. Refer to standard statistics textbooks;

[0297] (15) Apache RocketMQ: Distributed messaging platform, documentation: https: / / rocketmQ.apache.org / docs / .

[0298] The embodiment also provides an intelligent data stream processing implementation method based on Flink, comprising:

[0299] Intercepting business entity changes through the life cycle annotation of the JPA entity listener, triggering the recursive lazy loading algorithm to generate pre-change snapshots and post-change snapshots, then filtering sensitive fields after analyzing the @FieldMetadata annotation, binding the unique identifier of the business entity change event spliced by the entity primary key, class name, and SnowFlake algorithm, and obtaining a standardized message body, wherein @FieldMetadata is a field metadata annotation;

[0300] Parallelly pushing the standardized message body to the business queue for downstream service consumption and to the Data-Flow queue for Flink stream processing engine layer processing through a dedicated thread pool;

[0301] The Data-Flow queue event triggers the Flink engine to dynamically construct the DSL configuration JobGraph pipeline, and the CEP complex event detection and rolling window aggregation operation are synchronously performed during the execution process, wherein the window aggregation operation calls the RocksDB state backend to store the result and activates the TTL cleaning mechanism, and meanwhile, the load index is monitored in real time to dynamically adjust the TaskManager resource allocation, and the TaskManager is the smallest execution unit of the Flink job;

[0302] Checkpoint snapshot is periodically performed on the operator state during the stream processing, and if the back pressure signal is detected, the Kafka source end intake rate is reversely regulated and controlled;

[0303] The CEP engine detection and window aggregation result are input into the rule engine, statistical model and machine learning detector, the abnormal score is calculated according to the context library, and the real-time blocking, manual review or risk marking is triggered according to the grading threshold, and the treatment result is fed back to the rule library and model optimization module to realize closed-loop learning.

[0304] The application realizes multi-source data access through unified event model and pluggable connector, adopts zero-intrusion data-driven technology to automatically capture changes, guarantees asynchronous decoupling, end-to-end tracking and Exactly-Once semantics of data processing through double-channel message queue and full-link tracking technology, and effectively solves the problems of state expansion and abnormal early warning based on Flink state management and multi-level abnormal detection system, and improves the stability and intelligent level of the system.

[0305] At the data access level, the application realizes unified access and standardized processing of multi-source heterogeneous data, reduces the complexity of data integration, realizes real-time capture of data changes based on non-intrusive data-driven JPA entity listening without modifying business code, and supports various data source types and has good expansibility and flexibility.

[0306] At the processing efficiency level, the application realizes real-time processing of data based on the stream processing engine of Flink, greatly improves the timeliness of data processing, improves the processing efficiency and resource utilization of the system through distributed task scheduling and resource optimization, reduces system delay and improves throughput through asynchronous message processing and batch optimization.

[0307] At the reliability level, the application realizes the Exactly-Once semantics of data processing based on Flink Checkpoint and state management, ensures that data is not lost or repeated, ensures the stability and fault tolerance of the system through distributed high-availability architecture, and improves the recovery ability of the system under abnormal conditions through multi-level fault recovery mechanism.

[0308] In the intelligent level, multi-level anomaly detection strategy improves the identification ability of data anomaly; intelligent processing based on rule engine, statistical analysis and machine learning supports intelligent analysis of complex business scenarios; self-learning anomaly mode library continuously optimizes the effect of anomaly detection.

[0309] In the availability level, flexible visualization framework provides intuitive data insight and analysis ability; unified API interface facilitates integration with other systems; automatic alarm and notification mechanism improves the availability and operation and maintenance efficiency of the system.

[0310] In the development efficiency level, componentization and plug-in design support rapid development and system expansion; unified development interface and framework reduce the development complexity; declarative configuration mode reduces the code amount and improves the development efficiency.

[0311] The application realizes unified access of multi-source heterogeneous data, zero-intrusion data driving and multi-level anomaly detection, and significantly improves the real-time performance, reliability and intelligent level of the system. Through the application, enterprises can build a unified real-time data flow and analysis platform, realize real-time monitoring, state tracking and intelligent analysis of business data, and provide strong support for enterprise decision-making.

[0312] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.< / snowflakeid> < / snowflakeid> < / object>

Claims

1. An intelligent data stream processing system based on Flink, characterized in that: include: The data source access layer is configured with pluggable connectors to connect to multiple types of data sources, achieving unified data access and standardized conversion; The entity monitoring layer automatically captures business entity change events non-invasively and generates standardized message bodies based on the JPA entity monitoring mechanism, and asynchronously delivers them to dual channels. JPA is the Java Persistence Application Programming Interface. The unified event model layer is used to generate standardized events based on the standardized message body mapped by the data source access layer and route them to the Flink stream processing engine layer after expansion; The Flink stream processing engine layer is used for complex event processing, state management, window calculations, stream table transformations, and declarative pipeline definition and dynamic construction. Flink is the Apache Flink stream processing framework. Anomaly detection layer, which detects anomalies in data streams based on rule engines, statistical models, and machine learning algorithms; The distributed task scheduling layer schedules tasks based on the DAG dependency model and achieves load balancing through hotspot splitting and state migration. DAG is a directed acyclic graph. The result storage and visualization layer is used to store data using a hot and cold separation strategy and provide a dynamic visualization interface.

2. The Flink-based intelligent data stream processing system according to claim 1, characterized in that: The data source access layer includes data source connectors, data converters, and metadata managers; The data source connector is used to provide a standardized data source connection interface; The data source connector uses a standardized interface to define the core interface design, which includes initializing the connector, creating a Flink data source, and closing the connector; The data converter is used to convert the original data into a standard format within the system; The metadata manager is used to manage metadata information of a data source.

3. The Flink-based intelligent data stream processing system according to claim 1, characterized in that: After capturing business entity change events based on JPA lifecycle annotations, the entity monitoring layer triggers a recursive lazy loading algorithm, detects uninitialized fields through the Hibernate API, recursively loads related objects with a depth-first strategy and eliminates circular references, synchronously generates pre-change snapshots and post-change snapshots, and then performs field filtering; obtains a global TraceId based on the business entity primary key, business entity class name, and SnowFlake algorithm; constructs a standardized message body based on the global TraceId and the pre-change snapshot and post-change snapshot after field filtering, and asynchronously delivers it to the dual channels, where the global TraceId is a unique identifier for tracking request processes in a distributed system, the SnowFlake algorithm is a snowflake algorithm, Hibernate is a Java object-relational mapping framework, and API is an application program interface.

4. The Flink-based intelligent data stream processing system according to claim 3, characterized in that: The dual channels include a business queue and a Data-Flow queue. The business queue is used for asynchronous consumption by downstream microservices to achieve system decoupling. The Data-Flow queue is directly connected to the Flink stream processing engine layer to support real-time stream processing and is physically isolated from the business decoupling queue.

5. The Flink-based intelligent data stream processing system according to claim 1, characterized in that: The unified event model layer uses a hierarchical namespace to design event types and binds a dual-mode schema; an event converter parses the data source characteristics of the standardized message body to generate standardized business events; an event enhancer associates external context to expand the dimensions of the standardized business events; and based on the type and corresponding metadata of the standardized business events, the expanded standardized business events are sent to the Flink stream processing engine layer through an event router.

6. The Flink-based intelligent data stream processing system according to claim 1, characterized in that: The Flink stream processing engine layer includes a pipeline construction unit, an event pattern detection unit, a state calculation unit, a data enhancement unit, and a fault tolerance unit; The pipeline construction unit is used to parse the DSL configuration and trigger the FlowBuilder interface to dynamically build the JobGraph. The DSL configuration is the configuration of the Flink job, the FlowBuilder interface is the flow orchestration interface, and the JobGraph is the job topology graph. The event pattern detection unit is used to scan the event stream through the CEP engine, match the preset complex pattern, and trigger an alarm or aggregation operation, wherein the CEP engine is a complex event processing engine; The state calculation unit is used to call the RocksDB state backend to store intermediate results when performing window aggregation, and synchronously activate TTL to clean up expired states; where TTL is a time-to-live mechanism; RocksDB is an embedded key-value storage database; The data enhancement unit is used to associate external dimension table extended event attributes; The fault-tolerant unit ensures data consistency during fault recovery through a fault-tolerant mechanism based on checkpoints and savepoints; Checkpoint and Savepoint are both mechanisms in the Flink stream processing framework. Checkpoint is the periodic automatic saving of the status of stream processing jobs in Flink, while Savepoint is the manual triggering of saving the status of stream processing jobs in Flink.

7. The Flink-based intelligent data stream processing system according to claim 1, characterized in that: The anomaly detection layer includes a multi-dimensional parallel detection unit, a context-aware evaluation unit, and an early warning and response unit; The multi-dimensional parallel detection unit is used to obtain the rule matching results, indicator deviation scores and abnormal probability scores and input them into the context perception evaluation unit; The context-aware evaluation unit is configured to obtain an anomaly score with a context label based on the rule matching result, the indicator deviation score, and the anomaly probability score; The early warning and response unit performs graded early warning and response according to the anomaly score with context tags, and feeds back the handling results to the anomaly pattern library.

8. The Flink-based intelligent data stream processing system according to claim 1, characterized in that: The distributed task scheduling layer includes a resource perception unit, an intelligent decision-making unit, a flexible execution unit and a fault self-healing unit; The resource sensing unit is used to report the CPU, memory, back pressure and data skew of each node in real time through the monitor, and aggregate them into a load profile through the resource manager; The intelligent decision-making unit is used to call the load-aware algorithm based on the load profile through the task scheduler to calculate the optimal parallelism and resource quota that meets the DAG dependency model. If a hot task is found, it triggers splitting and generates a subtask topology; The elastic execution unit and cluster manager distribute the decisions of the intelligent decision-making unit through ZooKeeper, adjust resource allocation and migrate status online, and complete version rolling updates. ZooKeeper is a distributed coordination service. The fault self-healing unit is used to automatically activate the standby node through the fault recovery manager when an abnormality occurs at the process, node or cluster level, restore the operation status based on the latest savepoint, and reschedule the task immediately.

9. The Flink-based intelligent data stream processing system according to claim 1, characterized in that: The result storage and visualization layer includes a result writing unit, a lifecycle migration unit, a real-time query unit, a dynamic visualization unit and an alarm unit; The result writing unit processes each output of the Flink stream processing engine layer through the Sink to obtain the event heat; the storage manager assigns a label to the event according to the event heat and performs multi-level storage based on the label; wherein the Sink is a receiver operator; The lifecycle migration unit is used to automatically migrate data when the Redis memory usage or ClickHouse partition age triggers a threshold, and synchronize the new partition metadata to the SchemaRegistry after the migration is completed; at the same time, a resource release or application signal is sent to the resource manager of the distributed task scheduling layer through the API gateway, where Redis is a Redis in-memory database, ClickHouse is a ClickHouse column database, and SchemaRegistry is a pattern registration center; The real-time query unit retrieves the query result cache according to the query request, and if there is a miss, generates an optimal execution plan by using multi-dimensional indexing and partition pruning, performs parallel scanning and aggregation on multi-level storage data, and returns the aggregated result; The dynamic visualization unit, after receiving the aggregated results returned by the real-time query unit, automatically matches the display format from the visualization template library based on the data dimensions and features; the interaction logs generated by user operations are written back to the anomaly pattern library in real time to provide incremental samples for model retraining of the anomaly detection layer; The alarm unit calls the alarm intelligent grouping and merging algorithm based on the comprehensive anomaly score to perform multi-channel hierarchical push.

10. A method for implementing intelligent data stream processing based on Flink, characterized in that: include: Intercept business entity changes through the JPA entity listener's lifecycle annotations, triggering a recursive lazy loading algorithm to generate pre-change and post-change snapshots. Then, after parsing the @FieldMetadata annotation to filter sensitive fields, it binds the unique identifier of the business entity change event, which is composed of the entity primary key, class name, and the SnowFlake algorithm, to obtain a standardized message body. @FieldMetadata is the field metadata annotation. A dedicated thread pool is used to push standardized message bodies in parallel to business queues for consumption by downstream services and to Data-Flow queues for processing by the Flink stream processing engine layer. Data-Flow queue events trigger the Flink engine to dynamically build a DSL-configured JobGraph pipeline. During execution, complex event detection and rolling window aggregation operations are performed simultaneously. The window aggregation operation uses the RocksDB state backend to store the results and activates TTL. It also monitors load metrics in real time to dynamically adjust task manager resource allocation. During the stream processing process, checkpoint snapshots are periodically executed to persist the operator state. If backpressure signals are detected, the Kafka source ingestion rate is reversed. Integrate CEP engine detection and window aggregation results into the rule engine, statistical model and machine learning detector, associate the context library to calculate the anomaly score, and then trigger real-time blocking, manual review or risk marking according to the classification threshold. Feedback the disposal results to the rule library and model optimization module to achieve closed-loop learning.

Images