High-performance AIOT real-time data processing method and device based on Flink, Kafka and Dores

Abstract

The invention discloses a high-performance AIOT real-time data processing method and device based on Flink, Kafka and Dores, and the method comprises the steps: constructing an end-to-end hierarchical architecture, capturing multi-source database change data in real time through Flink CDC, and analyzing semi-structured data in combination with Apache NiFi, and generating a standardized data stream; the method comprises the following steps: realizing message routing based on KafkaTopic division of a service domain, ensuring the consistency of a data structure by using Schema Registry, and optimizing the transmission efficiency through a dynamic compression algorithm; an Flink real-time computing engine is adopted to implement window aggregation, asynchronous dimension table association and incremental state persistence strategies, and the resource fluctuation and state management problems in streaming computing are solved; time pre-partitioning and intelligent cold and hot layering are achieved through a Dores storage layer, the query performance is improved in combination with materialized views and multi-level indexes, and finally a full-link closed loop of collection-transmission-calculation-storage-management is formed. According to the invention, the problems that the real-time performance and the consistency are difficult to balance and the system complexity is high in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of AIOT data processing technology, and specifically to a high-performance AIOT real-time data processing method and device based on Flink, Kafka, and Doris. Background Art

[0002] With the rapid adoption of Artificial Intelligence for the Internet of Things (AIOT) technology, the amount of data generated by smart devices has exploded, with daily data throughput exceeding tens of billions. Hybrid data processing solutions, such as the Lambda and Kappa architectures, are now widely adopted in the industry. By integrating streaming computing components like Apache Flink and Kafka with batch processing frameworks like Hadoop and Spark, they have, to a certain extent, met the fundamental requirements for real-time data analysis. In recent years, the open source community has continuously iterated on technologies such as stream-batch integration and state management. The maturity of tools like Flink SQL and the Doris MPP engine has provided new technical paths for building real-time data warehouses, gradually reducing end-to-end latency from data collection to analysis from hours to minutes.

[0003] However, existing technical solutions still face multiple bottlenecks in practical implementation: traditional architectures struggle to ensure data consistency between the real-time and batch processing layers. The complex dual-link design leads to exponentially increased resource consumption and high operational and maintenance costs. The state management mechanism of the streaming computing engine is prone to memory overflows and computational jitter when dealing with high-frequency device data updates, seriously affecting system stability. The storage layer lacks optimization strategies for time-series data, and massive data writes and real-time query requests easily trigger I / O contention, causing a surge in query response latency. Furthermore, the patchwork combination of heterogeneous technology stacks makes data lineage difficult to track, requiring development teams to devote significant effort to resolving component compatibility issues, severely restricting the efficiency of releasing the value of real-time data in AIOT scenarios.

[0004] Therefore, how to invent and develop a high-performance AIOT real-time data processing method that can solve the problems of existing technologies such as the difficulty in balancing data real-time and consistency, high maintenance costs, and low resource utilization efficiency has become an urgent problem that needs to be solved. Summary of the Invention

[0005] To this end, this paper provides a high-performance AIOT real-time data processing method and device based on Flink, Kafka, and Doris. By building an end-to-end integrated real-time data processing architecture, unifying stream and batch processing links, reducing complexity and resource consumption, and ensuring the stability of high-frequency data computing through dynamic state compression and incremental persistence. Time series pre-partitioning and intelligent hot and cold tiering strategies optimize storage I / O efficiency, and standardized data routing connects lineage links, reducing adaptation costs and efficiently releasing the real-time value of AIOT data.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a high-performance AIOT real-time data processing method based on Flink, Kafka, and Doris, comprising:

[0007] Capture multi-source change data in real time through Flink CDC; process the semi-structured data in the multi-source change data through Apache NiFi to obtain a data stream in a unified data format;

[0008] Divide the data stream in the unified data format into the corresponding Topic of Kafka according to the business domain; perform structural verification on the JSON / Avro format data through SchemaRegistry, and generate a standardized message stream by setting a compression algorithm;

[0009] Perform real-time calculations on the standardized message stream using Flink and output calculation results;

[0010] The calculation results are written to the storage layer through Doris's ROUTINE LOAD task; a pre-partitioned table is created by the time field and the number of buckets is set, and hot and cold data are stored in a tiered manner.

[0011] As a preferred solution for high-performance AIOT real-time data processing based on Flink, Kafka, and Doris, in the process of real-time capture of multi-source change data through Flink CDC:

[0012] For relational databases, Flink CDC's Debezium engine captures INSERT, UPDATE, and DELETE events and converts them into change log streams identified by op_type.

[0013] For TCP / UDP protocol data on the device side, a custom collector is used to parse the protocol identifier in the message header, extract the payload, and filter outliers on the edge side.

[0014] For API data sources, configure the OAuth2.0 token refresh mechanism in NiFi to obtain long-term data through dynamic tokens.

[0015] As a preferred solution for the high-performance AIOT real-time data processing method based on Flink, Kafka, and Doris, in the process of dividing the data stream in the unified data format into the corresponding Kafka Topic according to the business domain:

[0016] An intelligent routing module is deployed on the Kafka producer side to dynamically select a Topic partition based on the hash value of the device_type field in the data stream of the unified data format.

[0017] As a preferred solution for the high-performance AIOT real-time data processing method based on Flink, Kafka, and Doris, in the process of performing real-time calculations on the standardized message stream through Flink:

[0018] Use Flink to perform aggregation calculations on rolling windows, sliding windows, and session windows; use LookUp Join to associate dimension table data; use Interval Join to associate time-constrained stream tables; use Temporal Join to associate data flow domain versioned tables; and implement an incremental checkpoint mechanism based on the RocksDB State Backend to maintain computing status.

[0019] As a preferred solution for high-performance AIOT real-time data processing based on Flink, Kafka, and Doris, in the process of tiered storage of hot and cold data:

[0020] For the hot data layer, three copies are used and memory cache is enabled for acceleration.

[0021] For the warm data layer, a two-copy mechanism is used and memory cache is disabled;

[0022] For the cold data tier, data is transferred to object storage and a single copy is retained.

[0023] As a preferred solution for high-performance AIOT real-time data processing based on Flink, Kafka, and Doris, the Dinky platform provides visual monitoring of the entire chain of AIOT real-time data processing; dynamically adjusts the parallelism of Flink jobs based on the results of data lineage analysis; and builds strategies based on Doris materialized views to form a closed-loop optimization mechanism for collection, transmission, calculation, and storage.

[0024] The present invention also provides a high-performance AIOT real-time data processing device based on Flink, Kafka, and Doris, including:

[0025] The data acquisition module is used to capture multi-source change data in real time through Flink CDC; the semi-structured data in the multi-source change data is processed through Apache NiFi to obtain a data stream in a unified data format;

[0026] The data processing and transmission module is used to divide the data stream in the unified data format into the corresponding Topic of Kafka according to the business domain; perform structural verification on the JSON / Avro format data through SchemaRegistry, and generate a standardized message stream by setting a compression algorithm;

[0027] A data computing module is used to perform real-time computing on the standardized message stream through Flink and output the computing results;

[0028] The data storage module is used to write the calculation results to the storage layer through Doris's ROUTINE LOAD task; create a pre-partitioned table by time field and set the number of buckets, and perform tiered storage for hot and cold data.

[0029] As a preferred solution for a high-performance AIOT real-time data processing device based on Flink, Kafka, and Doris, in the data acquisition module, during the process of capturing the multi-source change data in real time through Flink CDC:

[0030] For relational databases, Flink CDC's Debezium engine captures INSERT, UPDATE, and DELETE events and converts them into change log streams identified by op_type.

[0031] For TCP / UDP protocol data on the device side, a custom collector is used to parse the protocol identifier in the message header, extract the payload, and filter outliers on the edge side.

[0032] For API data sources, configure the OAuth2.0 token refresh mechanism in NiFi to obtain long-term data through dynamic tokens.

[0033] As a preferred solution for a high-performance AIOT real-time data processing device based on Flink, Kafka, and Doris, in the data processing and transmission module, when dividing the data stream in the unified data format into corresponding Kafka topics according to business domains:

[0034] An intelligent routing module is deployed on the Kafka producer side to dynamically select a Topic partition based on the hash value of the device_type field in the data stream of the unified data format.

[0035] As a preferred solution for a high-performance AIOT real-time data processing device based on Flink, Kafka, and Doris, in the data calculation module, during the real-time calculation of the standardized message stream using Flink:

[0036] Use Flink to perform aggregation calculations on rolling windows, sliding windows, and session windows; use LookUp Join to associate dimension table data; use Interval Join to associate time-constrained stream tables; use Temporal Join to associate data flow domain versioned tables; and implement an incremental checkpoint mechanism based on the RocksDB State Backend to maintain computing status.

[0037] As a preferred solution for a high-performance AIOT real-time data processing device based on Flink, Kafka, and Doris, in the data storage module, during the tiered storage of hot and cold data:

[0038] For the hot data layer, three copies are used and memory cache is enabled for acceleration.

[0039] For the warm data layer, a two-copy mechanism is used and memory cache is disabled;

[0040] For the cold data tier, data is transferred to object storage and a single copy is retained.

[0041] As the preferred solution for high-performance AIOT real-time data processing devices based on Flink, Kafka, and Doris, the Dinky platform provides visual monitoring of the entire chain of AIOT real-time data processing; dynamically adjusts the parallelism of Flink jobs based on the results of data lineage analysis; and builds strategies based on Doris materialized views to form a closed-loop optimization mechanism for collection, transmission, calculation, and storage.

[0042] This invention has the following advantages: it uses Flink CDC to capture multi-source change data in real time; uses Apache NiFi to process the semi-structured data within this multi-source change data to obtain a data stream in a unified data format; partitions this data stream into corresponding Kafka topics by business domain; uses SchemaRegistry to perform structural verification on JSON / Avro formatted data and generates standardized message streams by configuring a compression algorithm; uses Flink to perform real-time calculations on this standardized message stream and output the calculation results; and writes these calculation results to the storage layer using the Doris ROUTINE LOAD task; creates a pre-partitioned table based on the time field, sets the number of buckets, and implements tiered storage for hot and cold data. This invention reduces latency from data acquisition to result presentation to seconds to minutes, providing an exceptional user experience. It supports high-throughput, low-latency data processing, meeting the requirements for real-time analysis of massive amounts of data in AIOT scenarios. It ensures data consistency and accuracy, providing a reliable basis for business decision-making. It also offers excellent scalability and fault tolerance, supporting continued business growth. The present invention reduces system complexity, improves development efficiency and resource utilization; the present invention constructs standardized, reusable technical components to reduce the implementation costs of subsequent projects. The present invention effectively unifies the stream batch processing link by constructing an end-to-end integrated real-time data processing architecture, significantly reduces system complexity and resource redundancy consumption, and at the same time ensures computing stability in high-frequency data update scenarios based on dynamic state compression and incremental persistence mechanisms. Through the storage layer time-series pre-partitioning and intelligent hot and cold tiering strategies, the problem of I / O resource contention is greatly alleviated, and the real-time query response efficiency is improved. In addition, the deeply integrated technical components and standardized data routing system completely connect the data lineage link, simplifying the compatibility and adaptation costs between heterogeneous systems, allowing the real-time value of massive AIOT device data to be quickly released, and providing highly reliable support for intelligent decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0044] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0045] Figure 1 This is a flow chart of the high-performance AIOT real-time data processing method based on Flink, Kafka, and Doris provided in Example 1 of the present invention;

[0046] Figure 2 This is a schematic diagram comparing the specific data flows of the present invention and the prior art in the high-performance AIOT real-time data processing method based on Flink, Kafka, and Doris provided in Example 1 of the present invention;

[0047] Figure 3 This is a schematic diagram of the architecture of a high-performance AIOT real-time data processing device based on Flink, Kafka, and Doris provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0048] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0049] Example 1

[0050] See also Figure 1 , Embodiment 1 of the present invention provides a high-performance AIOT real-time data processing method based on Flink, Kafka, and Doris, comprising the following steps:

[0051] S1. Capture multi-source change data in real time through Flink CDC; process the semi-structured data in the multi-source change data through Apache NiFi to obtain a data stream in a unified data format;

[0052] S2. Divide the data stream in the unified data format into corresponding topics in Kafka according to business domains; perform structural verification on the JSON / Avro format data through Schema Registry, and generate a standardized message stream by setting a compression algorithm;

[0053] S3. Perform real-time calculations on the standardized message stream using Flink and output calculation results.

[0054] S4. Write the calculation results to the storage layer through Doris's ROUTINE LOAD task; create a pre-partitioned table by time field and set the number of buckets, and perform tiered storage for hot and cold data.

[0055] In this embodiment, in step S1, multi-source change data is captured in real time through Flink CDC; semi-structured data in the multi-source change data is processed through Apache NiFi to obtain a data stream in a unified data format;

[0056] Specifically, Flink CDC is used to capture the change data streams of relational databases such as MySQL and PostgreSQL in real time, and the Debezium engine is used to parse binlog logs and convert them into standardized change events (including INSERT / UPDATE / DELETE operations), supporting breakpoint resumption and exactly-once semantic guarantees; at the same time, through the visual data stream orchestration function of Apache NiFi, it connects to API interfaces, log files and semi-structured data reported by edge devices to perform data cleaning, such as (filling in empty values, removing outliers) and format conversion (such as XML / CSV to JSON); for specific industrial protocol (such as Modbus, OPC UA) devices, lightweight collectors are deployed for protocol parsing and edge-side preprocessing, such as data downsampling and threshold alarms, and finally output a standardized data stream that uniformly contains timestamps, device IDs, and business fields to ensure time alignment and format consistency of multi-source data.

[0057] In this embodiment, in step S2, the data stream in the unified data format is divided into corresponding topics of Kafka according to the business domain; the structure of the JSON / Avro format data is verified through the Schema Registry, and a standardized message stream is generated by setting a compression algorithm;

[0058] Specifically, based on business domains (such as device monitoring and environmental perception) and data types (such as status data and log data), standardized data streams are routed to the corresponding Kafka topics, and a composite naming rule of "business line + data category" (such as iot_env_sensor) is used to achieve logical isolation. By integrating the Schema Registry, strong type verification is implemented for JSON / Avro format data, messages that do not conform to the predefined schema are rejected, and abnormal data is recorded in the Dead Letter Queue for subsequent analysis.

[0059] The compression algorithm is dynamically selected based on the characteristics of the data type. LZ4 is used for text data and Snappy is used for binary data. The compression ratio is controlled at 30%-50%. At the same time, end-to-end CRC verification and message idempotence production mechanism are enabled to ensure that there is no loss or duplication during data transmission. For bursty traffic, horizontal expansion is achieved through Kafka's dynamic partition expansion strategy (triggered by minute-level message volume thresholds), supporting a TPS throughput of millions.

[0060] In this embodiment, in step S3, the standardized message stream is calculated in real time by Flink, and the calculation result is output;

[0061] Specifically, the following calculations are performed on the standardized message stream:

[0062] 1. Data parsing and window aggregation:

[0063] Event time semantics are used to parse the timestamp fields in the message stream, and rolling windows (1 minute to 1 hour) and sliding windows (1 minute step) are configured based on business rules. Local-Global dual aggregation optimization is implemented for device status data, first pre-aggregating in local buckets, and then globally merging and outputting the data to reduce network transmission overhead.

[0064] 2. Flow table association and status update:

[0065] Asynchronously access external dimension tables through Async I / O, and use Guava Cache to build an LRU local cache (cache hit rate > 80%); implement Temporal Join on device metadata to match dimension snapshots within the valid time range based on the data version number.

[0066] 3. State persistence and fault tolerance:

[0067] Configure the RocksDB state backend to persist the window aggregate state to HDFS, with a checkpoint interval of 1-5 minutes. Implement an incremental checkpoint mechanism to synchronize only changed SST files, reducing snapshot generation time by over 40%. Set a state TTL (Time-To-Live) policy to automatically clean up offline device data (retain data for 7 days by default).

[0068] 4. Dynamic resource control:

[0069] The number of TaskManager slots is automatically adjusted based on back pressure monitoring, and a 2x parallelism expansion is triggered when the message delay exceeds 5 seconds. KeyBy+rebalance dual partitioning is implemented for data skew streams, and hot device IDs are identified through pre-sampling and routed separately.

[0070] In this embodiment, in step S4, the calculation results are written to the storage layer through Doris's ROUTINE LOAD task; a pre-partitioned table is created according to the time field and the number of buckets is set, and hot and cold data are stored in a tiered manner.

[0071] Specifically, Doris's ROUTINE LOAD task consumes calculation result data from Kafka, creates a dynamic pre-partitioned table based on the event time field (event_time) (partitioned by day, with partitions created 400 days before and after in advance), sets the number of buckets based on the data size (500MB-1GB per bucket, 32 buckets by default), and enables the enable_unique_key_merge_on_write feature to achieve real-time deduplication and update of primary keys;

[0072] Implement hot and cold tiering strategies for storage tiers:

[0073] For the hot data layer (nearly 7 days): stored on SSD disks, using a three-copy mechanism and enabling memory cache acceleration (cache TTL = 1 hour);

[0074] For the warm data layer (7-90 days): stored on HDD disks, 2 copies and memory cache is disabled;

[0075] For the cold data layer (90 days ago): transfer to the S3 object storage, retain one copy and enable ZSTD compression (compression ratio ≥ 5:1);

[0076] In this embodiment, a materialized view is constructed to pre-aggregate high-frequency query indicators (such as hourly device online rate), an inverted index is established for the timestamp field, and a Bitmap index is established for the numeric field. Combined with the query result cache (cache hit rate > 85%), sub-second response is achieved, and data compaction is performed regularly (daily at dawn) to optimize file merging and storage fragmentation cleanup.

[0077] In this embodiment, the entire link of steps S1-S4 is visually monitored through the Dinky platform, and the Flink job parallelism and Doris materialized view construction strategy are dynamically adjusted according to the results of data lineage analysis, forming a closed-loop optimization mechanism of collection → transmission → calculation → storage.

[0078] The visual monitoring of the entire link of steps S1-S4 includes:

[0079] System resource layer: Real-time tracking of infrastructure indicators such as cluster CPU / memory usage (threshold alarm: >85% for 5 minutes), disk IO throughput (read and write latency >100ms triggers an alarm), and network bandwidth usage (>70% for 10 minutes).

[0080] Application component layer: Flink jobs: monitor backpressure ratio (alarm when Backpressure Ratio > 30%), checkpoint success rate (trigger troubleshooting when < 95%), and operator processing latency (alarm when P99 > 5 seconds).

[0081] Kafka: Real-time display of Topic message backlog (alarm if Lag > 100,000 messages) and partition balance (alarm if the difference between the maximum and minimum partition message volumes is > 3 times);

[0082] Doris: Monitors query response time (P95 > 1 second alarm), compaction progress (number of unfinished compaction files > 100 alarm), and BE node load balance (data distribution deviation > 20% alarm).

[0083] Business indicator layer: data quality score (based on indicators such as null value rate and format error rate, with a score < 80 points triggering an alarm); end-to-end processing delay (SLA delay from data collection to queryability > 8 seconds triggering an alarm); abnormal event triggering frequency (such as the number of device offline alarms / minute exceeding the threshold).

[0084] In this embodiment, the parallelism of Flink jobs is dynamically adjusted based on the data lineage analysis results;

[0085] Specifically, by parsing the Flink SQL execution plan, KafkaTopic subscription relationship, and Doris table write path, it automatically generates a data lineage graph (stored in the Neo4j graph database) and records field-level lineage relationships (such as tracing the dwd_iot.event_time field back to Kafka env_sensor.timestamp). When dynamic KafkaTopic policy adjustments are detected, lineage impact analysis is supported:

[0086] Based on the back pressure rate and latency indicators, the system automatically adjusts the degree of parallelism (for example, expanding the env_sensor processing job from 32 concurrent users to 64 concurrent users). After identifying hot keys (such as high-frequency device IDs), it dynamically injects salt values ​​(Salting) and repartitioning logic to alleviate data skew. Based on the window calculation load (for example, when the rolling window aggregate CPU usage is >70%), it automatically switches to the Local-Global optimization mode.

[0087] In this embodiment, a Doris materialized view is constructed. By monitoring high-frequency query patterns (such as Top 10 queries), matching materialized views (such as mv_device_1h_summary) are automatically created, and inefficient views are eliminated (automatically deleted if they have not been accessed for 7 days). Based on the popularity of data access (such as the query frequency in the last hour), the hot and cold tiering strategy is dynamically adjusted (such as temporarily promoting frequently accessed warm data to the hot layer). When an uneven bucket data volume is detected (the difference between the maximum and minimum bucket data volumes is greater than 2 times), a bucket merge / split operation is triggered. When the schema changes, the Flink jobs and Doris materialized views that depend on the topic are automatically marked, and a change impact assessment report is pushed.

[0088] In this embodiment, the Dinky platform also has a closed-loop feedback control mechanism;

[0089] Specifically, the automated rule engine triggers adjustment actions (such as Flink scaling) based on monitoring indicators. After execution, it continuously monitors key indicators (such as latency reduction rate and CPU usage changes). If the expected effect is not achieved (such as latency reduction <20%), it rolls back to the previous stable configuration and triggers manual intervention.

[0090] By establishing an AB testing mechanism, we can retain both old and new versions of jobs for key strategies (such as window size adjustment), compare processing efficiency (such as throughput differences), and select the optimal solution.

[0091] Successful optimization strategies (such as salt value allocation rules for specific data skew scenarios) are stored in the knowledge base and automatically recommended for subsequent similar scenarios. LSTM models are trained based on historical monitoring data to predict resource bottlenecks (such as the peak Kafka throughput in the next hour) and trigger elastic expansion in advance.

[0092] In this embodiment, Figure 2 As shown in the figure, by comparing the existing technology with the specific data flow of the present invention, it can be seen that the present invention decouples the data extraction layer from the computing layer by splitting the original single-point Flink CDC into independent Flink clusters. The newly added streaming computing layer supports complex ETL operations and streaming Lookup Join. Combined with Kafka's closed-loop write-back mechanism, real-time indicators can be fed back to the business system, and the stream processing throughput is increased by 5-10 times.

[0093] To address the read-write contention issues inherent in a single Doris cluster, a multi-cluster read-write separation design was adopted: a high-frequency write cluster focused on raw data access, while an OLAP query cluster handled analytical workloads. The Doris vectorization engine optimized LookupJoin operations, reducing memory consumption by 60% and increasing OLAP query response speed by 300%.

[0094] This invention upgrades key near-real-time links (such as NiFi-MongoDB asynchronous transmission) to real-time direct connection channels, and cooperates with Doris multi-active cluster deployment to achieve cross-segment data sharding and parallel query, compressing end-to-end data delay from minutes to 50 milliseconds, meeting financial-grade real-time requirements.

[0095] This invention also incorporates Kubernetes containerized scheduling, enabling second-level elastic scaling of components like Flink and Doris. Multiple Doris clusters support dynamic load balancing, increasing burst traffic capacity by 10x and reducing resource utilization fluctuations by 70%. Furthermore, monitoring points for Kafka write-back links are improved, forming an observability system covering the entire link.

[0096] In one possible embodiment, an example of Shanghai Quzhi AIOT real-time data warehouse processing is provided as follows:

[0097] Shanghai Quzhi AIOT platform needs to process second-level data from more than 500,000 smart devices (including environmental sensors, industrial controllers, etc.) in real time, with an average daily data volume of more than 1 billion items. It requires end-to-end latency to be less than 5 seconds and supports business scenarios such as real-time monitoring of device status and environmental anomaly warning.

[0098] Specific processing steps:

[0099] T1. Multi-source heterogeneous data collection:

[0100] For relational data: Use Flink CDC to directly connect to devices to manage a MySQL cluster (16 nodes). Capture binlog logs of device configuration changes (such as threshold adjustments) and status updates (such as online / offline switching). Parse them into a structured event stream consisting of op_type (operation type), device_id, timestamp, and payload. Set the resumable transmission fault tolerance interval to 1 minute.

[0101] For edge device data: deploy a lightweight collector to parse Modbus TCP protocol messages, extract sensor IDs and temperature / humidity values, and perform outlier filtering on the edge (for example, marking temperatures > 150°C as invalid data).

[0102] For log and API data: Use NiFi to consume HTTP API JSON format device logs (5000+ requests per second), parse the log_level and error_code fields, standardize them into a unified schema, and then inject them into Kafka.

[0103] T2. Business domain message routing and compression:

[0104] Topic division: Divide topics by business type, for example:

[0105] equipment_status: equipment heartbeat and online status data (number of partitions = 16, compression algorithm = LZ4)

[0106] env_sensor: Environmental indicators such as temperature, humidity, and PM2.5 (number of partitions = 32, compression algorithm = Snappy)

[0107] Data verification: Integrates with Schema Registry to define the Avro schema of env_sensor (including 12 required fields), intercepting illegal data with missing location_id or values ​​out of range (such as humidity > 100%), and intercepting approximately 1.2 million abnormal data items per day.

[0108] Traffic control: When the Kafka producer throughput exceeds 50,000 messages per second, dynamic partition expansion is automatically enabled (adding 4 partitions at a time) to ensure that the message accumulation rate under peak traffic is less than 5%.

[0109] T3, Flink streaming computing optimization:

[0110] Window calculation: For environmental sensor data, a 5-minute rolling window is divided by event time to calculate the average temperature and humidity in each area (allowing 2 seconds of out-of-order data). Local-Global aggregation optimization is used to reduce network transmission by 60%.

[0111] Device status data: Define a 10-minute session window to count the device's continuous online time, with a timeout threshold of 3 minutes.

[0112] Dimension table association: Asynchronously query the device metadata database (Redis cluster) through Async I / O, cache the metadata of active devices in the last 24 hours (cache size = 2GB, hit rate > 90%), and achieve millisecond-level mapping of device_id to location_id.

[0113] State management: Configure RocksDB state backend (local SSD storage), checkpoint interval = 2 minutes, incremental snapshot size controlled within 50MB; automatically clean up device state data that has been offline for more than 7 days

[0114] Dynamic scaling: The number of TaskManagers is automatically adjusted based on back pressure rate monitoring. When the env_sensorTopic delay exceeds 10 seconds, the number of Task Slots is expanded from 32 to 64, with a peak processing capacity of 200,000 records per second.

[0115] T4, Doris storage and query acceleration:

[0116] Data storage: The ROUTINE LOAD task is used to write the Flink calculation results into the Doris table dwd_iot. The data is pre-partitioned by the event_time field (one partition per day, with partitions created 400 days before and after). The number of buckets is 64 (the data volume of a single bucket is ≈ 800 MB).

[0117] Hot and cold stratification:

[0118] Hot data (within 7 days): stored on NVMe SSD, 3 copies, memory cache TTL = 2 hours Warm data (7-90 days): transferred to HDD, 2 copies, enable ZSTD compression (compression ratio = 6:1)

[0119] Cold data (90 days+): Archived to Alibaba Cloud OSS, retaining one copy.

[0120] Query optimization: Create a materialized view mv_1h_avg to pre-calculate the hourly average of environmental indicators for each region, with query response time less than 300ms. Build a BloomFilter index for device_id, improving query efficiency by 8 times. Perform compaction to merge small files every morning, saving 40% of storage space.

[0121] In summary, this invention uses Flink CDC to capture multi-source change data in real time; uses Apache NiFi to process the semi-structured data within this multi-source change data, obtaining a data stream in a unified data format; partitions this data stream into corresponding Kafka topics by business domain; uses Schema Registry to perform structural verification on JSON / Avro formatted data and generates standardized message streams by configuring compression algorithms; uses Flink to perform real-time calculations on this standardized message stream and output the calculation results; and writes these calculation results to the storage layer using the Doris ROUTINE LOAD task; creates a pre-partitioned table based on the time field, sets the number of buckets, and implements tiered storage for hot and cold data. This invention reduces latency from data source acquisition to result presentation to seconds to minutes, providing an exceptional user experience. It supports high-throughput, low-latency data processing capabilities, meeting the real-time analysis requirements of massive data volumes in AIOT scenarios. It ensures data consistency and accuracy, providing a reliable basis for business decision-making. It also offers excellent scalability and fault tolerance, supporting sustained business growth. The present invention reduces system complexity, improves development efficiency and resource utilization; the present invention constructs standardized, reusable technical components to reduce the implementation costs of subsequent projects. The present invention effectively unifies the stream batch processing link by constructing an end-to-end integrated real-time data processing architecture, significantly reduces system complexity and resource redundancy consumption, and at the same time ensures computing stability in high-frequency data update scenarios based on dynamic state compression and incremental persistence mechanisms. Through the storage layer time-series pre-partitioning and intelligent hot and cold tiering strategies, the problem of I / O resource contention is greatly alleviated, and the real-time query response efficiency is improved. In addition, the deeply integrated technical components and standardized data routing system completely connect the data lineage link, simplifying the compatibility and adaptation costs between heterogeneous systems, allowing the real-time value of massive AIOT device data to be quickly released, and providing highly reliable support for intelligent decision-making.

[0122] It should be noted that the method of the embodiments of the present disclosure can be performed by a single device, such as a computer or server. The method of the embodiments of the present disclosure can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0123] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0124] Example 2

[0125] See also Figure 3 , Embodiment 2 of the present invention also provides a high-performance AIOT real-time data processing device based on Flink, Kafka, and Doris, including:

[0126] Data acquisition module 001 is used to capture multi-source change data in real time through Flink CDC; process the semi-structured data in the multi-source change data through Apache NiFi to obtain a data stream in a unified data format;

[0127] Data processing and transmission module 002 is used to divide the data stream in the unified data format into corresponding topics in Kafka according to business domains; perform structural verification on JSON / Avro format data through Schema Registry, and generate standardized message streams by setting compression algorithms;

[0128] Data calculation module 003, used to perform real-time calculations on the standardized message stream through Flink and output calculation results;

[0129] The data storage module 004 is used to write the calculation results into the storage layer through Doris's ROUTINE LOAD task; create a pre-partitioned table according to the time field and set the number of buckets, and perform tiered storage for hot and cold data.

[0130] In this embodiment, in the data acquisition module 001, in the process of capturing the multi-source change data in real time through Flink CDC:

[0131] For relational databases, Flink CDC's Debezium engine captures INSERT, UPDATE, and DELETE events and converts them into change log streams identified by op_type.

[0132] For TCP / UDP protocol data on the device side, a custom collector is used to parse the protocol identifier in the message header, extract the payload, and filter outliers on the edge side.

[0133] For API data sources, configure the OAuth2.0 token refresh mechanism in NiFi to obtain long-term data through dynamic tokens.

[0134] In this embodiment, in the data processing and transmission module 002, in the process of dividing the data stream in the unified data format into corresponding topics of Kafka according to business domains:

[0135] An intelligent routing module is deployed on the Kafka producer side to dynamically select a Topic partition based on the hash value of the device_type field in the data stream of the unified data format.

[0136] In this embodiment, in the data calculation module 003, during the process of performing real-time calculation on the standardized message stream through Flink:

[0137] Use Flink to perform aggregation calculations on rolling windows, sliding windows, and session windows; use LookUp Join to associate dimension table data; use Interval Join to associate time-constrained stream tables; use Temporal Join to associate data flow domain versioned tables; and implement an incremental checkpoint mechanism based on the RocksDB State Backend to maintain computing status.

[0138] In this embodiment, in the data storage module 004, during the process of tiered storage of hot and cold data:

[0139] For the hot data layer, three copies are used and memory cache is enabled for acceleration.

[0140] For the warm data layer, a two-copy mechanism is used and memory cache is disabled;

[0141] For the cold data tier, data is transferred to object storage and a single copy is retained.

[0142] In this embodiment, the Dinky platform is used to visually monitor the entire link of AIOT real-time data processing; the parallelism of Flink jobs is dynamically adjusted based on the results of data lineage analysis; and a strategy is built based on Doris materialized views to form a closed-loop optimization mechanism for collection, transmission, calculation, and storage.

[0143] It should be noted that the information interaction, execution process, etc. between the modules of the above-mentioned system are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and no further details will be given here.

[0144] Example 3

[0145] Example 3 of the present invention provides a non-transitory computer-readable storage medium, which stores program code for a high-performance AIOT real-time data processing method based on Flink, Kafka, and Doris. The program code includes instructions for executing Example 1 or any possible implementation thereof.

[0146] Computer-readable storage media can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0147] Example 4

[0148] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0149] The processor and the memory communicate with each other through a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the high-performance AIOT real-time data processing method based on Flink, Kafka and Doris of Example 1 or any possible implementation thereof.

[0150] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading software code stored in a memory. The memory can be integrated into the processor or located outside the processor and exist independently.

[0151] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode.

[0152] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing system. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Alternatively, they can be implemented using program code executable by a computing system, and thus, they can be stored in a storage system and executed by the computing system. In some cases, the steps shown or described herein can be performed in a different order than that shown, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0153] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A high-performance AIOT real-time data processing method based on Flink, Kafka, and Doris, characterized by: include: Capture multi-source change data in real time through Flink CDC; Processing the semi-structured data in the multi-source change data through Apache NiFi to obtain a data stream in a unified data format; Divide the data stream in the unified data format into the corresponding Topic of Kafka according to the business domain; perform structural verification on the JSON / Avro format data through SchemaRegistry, and generate a standardized message stream by setting a compression algorithm; Perform real-time calculations on the standardized message stream using Flink and output calculation results; The calculation results are written to the storage layer through Doris's ROUTINE LOAD task; a pre-partitioned table is created by the time field and the number of buckets is set, and hot and cold data are stored in a tiered manner.

2. The high-performance AIOT real-time data processing method based on Flink, Kafka and Doris according to claim 1 is characterized in that: During the real-time capture of multi-source change data using Flink CDC: For relational databases, Flink CDC's Debezium engine captures INSERT, UPDATE, and DELETE events and converts them into change log streams identified by op_type. For TCP / UDP protocol data on the device side, a custom collector is used to parse the protocol identifier in the message header, extract the payload, and filter outliers on the edge side. For API data sources, configure the OAuth2.0 token refresh mechanism in NiFi to obtain long-term data through dynamic tokens.

3. The high-performance AIOT real-time data processing method based on Flink, Kafka and Doris according to claim 2 is characterized in that: In the process of dividing the data streams in the unified data format into corresponding topics in Kafka according to business domains: An intelligent routing module is deployed on the Kafka producer side to dynamically select a Topic partition based on the hash value of the device_type field in the data stream of the unified data format.

4. The high-performance AIOT real-time data processing method based on Flink, Kafka and Doris according to claim 3 is characterized in that: During the real-time calculation of the standardized message stream using Flink: Use Flink to perform aggregation calculations on rolling windows, sliding windows, and session windows; use LookUp Join to associate dimension table data; use Interval Join to associate time-constrained stream tables; use Temporal Join to associate data flow domain versioned tables; and implement an incremental checkpoint mechanism based on the RocksDB State Backend to maintain computing status.

5. The high-performance AIOT real-time data processing method based on Flink, Kafka and Doris according to claim 4 is characterized in that: In the process of tiered storage of hot and cold data: For the hot data layer, three copies are used and memory cache is enabled for acceleration. For the warm data layer, a two-copy mechanism is used and memory cache is disabled; For the cold data tier, data is transferred to object storage and a single copy is retained.

6. The high-performance AIOT real-time data processing method based on Flink, Kafka and Doris according to claim 5 is characterized in that: The Dinky platform provides visual monitoring of the entire chain of AIOT real-time data processing; dynamically adjusts the parallelism of Flink jobs based on data lineage analysis results; and builds strategies based on Doris materialized views to form a closed-loop optimization mechanism for collection, transmission, calculation, and storage.

7. A high-performance AIOT real-time data processing device based on Flink, Kafka, and Doris, characterized by: include: The data collection module is used to capture multi-source change data in real time through Flink CDC; Processing the semi-structured data in the multi-source change data through Apache NiFi to obtain a data stream in a unified data format; The data processing and transmission module is used to divide the data stream in the unified data format into the corresponding Topic of Kafka according to the business domain; perform structural verification on the JSON / Avro format data through SchemaRegistry, and generate a standardized message stream by setting a compression algorithm; A data computing module is used to perform real-time computing on the standardized message stream through Flink and output the computing results; The data storage module is used to write the calculation results to the storage layer through Doris's ROUTINE LOAD task; create a pre-partitioned table by time field and set the number of buckets, and perform tiered storage for hot and cold data.

8. The high-performance AIOT real-time data processing device based on Flink, Kafka and Doris according to claim 7 is characterized in that: In the data acquisition module, during the process of capturing the multi-source change data in real time through Flink CDC: For relational databases, Flink CDC's Debezium engine captures INSERT, UPDATE, and DELETE events and converts them into change log streams identified by op_type. For TCP / UDP protocol data on the device side, a custom collector is used to parse the protocol identifier in the message header, extract the payload, and filter outliers on the edge side. For API data sources, configure the OAuth2.0 token refresh mechanism in NiFi to obtain long-term data through dynamic tokens.

9. The high-performance AIOT real-time data processing device based on Flink, Kafka and Doris according to claim 8 is characterized in that: In the data processing and transmission module, in the process of dividing the data stream in the unified data format into the corresponding topics of Kafka according to the business domain: An intelligent routing module is deployed on the Kafka producer side to dynamically select a Topic partition based on the hash value of the device_type field in the data stream of the unified data format.

10. The high-performance AIOT real-time data processing device based on Flink, Kafka and Doris according to claim 9 is characterized in that: In the data computing module, during the real-time computing of the standardized message stream by Flink: Use Flink to perform aggregation calculations on rolling windows, sliding windows, and session windows; use LookUp Join to associate dimension table data; use Interval Join to associate time-constrained stream tables; use Temporal Join to associate data flow domain versioned tables; and implement an incremental checkpoint mechanism based on the RocksDB State Backend to maintain computing status.

Images