Distributed big data real-time processing and analysis system

Abstract

The application discloses a distributed big data real-time processing and analysis system, belongs to the technical field of big data real-time processing, and comprises a multidimensional data stream acquisition module, an elastic resource scheduling module, an incremental state management module, an intelligent fault-tolerant coordination module and a real-time analysis output module. Fine dynamic scheduling of computing resources is realized through an adaptive weight adjustment strategy, resource utilization is improved by more than 40%, through fine-grained incremental state management technology, only changed data is recalculated, processing delay is reduced by 60%, throughput is improved by 150%, through an intelligent fault-tolerant coordination mechanism, fault recovery time is controlled within 200 ms, and availability reaches 99.99%. The problems of inflexible resource scheduling, low data processing efficiency and weak fault tolerance in the prior art are solved, efficient, low-delay and high-reliability processing of massive real-time data streams is realized, and the application is suitable for application scenarios such as financial risk control, telecommunication monitoring and Internet of Things analysis.

Description

Technical Field

[0001] This invention relates to the field of real-time big data processing technology, specifically to a distributed real-time big data processing and analysis system, and particularly to a real-time streaming data processing technology based on adaptive elastic resource scheduling and fine-grained incremental state management. Background Technology

[0002] Distributed big data real-time processing technology is a core supporting technology for current data-intensive applications, widely used in fields such as financial risk control, telecommunications network monitoring, IoT data analysis, social media stream analysis, and smart manufacturing. This technology uses a distributed computing architecture to process and analyze massive, continuously generated real-time data streams with low latency, providing real-time insights for business decisions. With the explosive growth of IoT devices and the widespread adoption of 5G networks, the rate of data generation is increasing exponentially. IDC predicts that the global data volume will reach 175 ZB by 2025, of which approximately 30% will be real-time data. This places higher demands on the throughput, latency, and scalability of real-time data processing systems.

[0003] Existing distributed real-time data stream processing systems are primarily built on open-source frameworks such as Apache Storm, Apache Flink, and Spark Streaming. These systems typically employ a master-slave architecture, with the master node responsible for task scheduling and resource allocation, and worker nodes responsible for the actual data processing tasks. When processing streaming data, the system divides the data stream into multiple data shards, distributes them to different computing nodes for parallel processing, and finally aggregates and outputs the processing results.

[0004] Referring to prior art document CN119356879A, this patent proposes a distributed real-time data stream processing system. It analyzes the switching behavior of data flow states through a state mapping modeling module to generate a data flow state mapping diagram. The system employs a dynamic load calculation module to extract resource usage and load range, generating node load analysis diagrams and node load prediction analysis results. The system also includes a node task adjustment module and a resource scheduling execution module for adjusting task allocation and resource configuration. While this technical solution achieves a certain degree of monitoring of data stream states and dynamic load prediction, it still has the following shortcomings:

[0005] First, regarding resource scheduling strategies, this technical solution employs a static load prediction method based on historical data, relying on predefined state switching patterns and fixed resource allocation rules. In practical applications, data flow arrival patterns and processing demands are highly dynamic and unpredictable, especially in scenarios involving sudden traffic spikes and abnormal events. Static scheduling strategies struggle to adapt quickly to changes in system state, leading to low resource utilization and increased response latency. Furthermore, this solution's resource scheduling is adjusted at the node level, lacking fine-grained control over computing resources. In mixed workload scenarios, it cannot flexibly allocate CPU, memory, and network bandwidth resources, easily resulting in resource fragmentation and localized resource contention.

[0006] Secondly, regarding data processing efficiency, this technical solution adopts a traditional full-scale computation model, performing complete computation on each incoming batch of data. In large-scale data stream scenarios, the repetitive computation of a large amount of data limits system throughput. A key issue commonly faced by modern stream processing systems is how to efficiently manage and update computational states. This solution lacks an effective incremental computation mechanism, failing to fully utilize the temporal locality of data and the reusability of computation, resulting in significant waste of computational resources. Especially in scenarios such as window aggregation, join operations, and iterative computation, the cost of full-scale computation increases linearly or even superlinearly with the increase in data volume, severely restricting the system's scalability.

[0007] Third, regarding fault tolerance and recovery mechanisms, while this technical solution can detect node failures and migrate tasks, its fault tolerance mechanism relies on coarse-grained task-level checkpoints, which incur significant overhead in creating and restoring checkpoints. In high-frequency failure scenarios, frequent checkpoint operations can significantly reduce the system's effective throughput. Furthermore, the solution's fault recovery process lacks consideration for task priorities and data dependencies, potentially leading to excessively long recovery delays for critical tasks in large-scale failure scenarios, impacting the system's quality of service (QoS) assurance. More importantly, this solution does not provide predictive fault detection or proactive fault tolerance mechanisms; it can only perform passive recovery after a failure occurs, failing to proactively mitigate potential system risks.

[0008] Fourth, regarding system availability, this technical solution has shortcomings in ensuring state consistency and distributed coordination. In a distributed environment, ensuring the consistency of computational states and the correctness of results is a challenge when multiple computing nodes concurrently process data streams. This solution lacks a mechanism to guarantee exact-once processing semantics, which may lead to data duplication or loss in scenarios such as node failure and network partitioning, affecting the accuracy of computation results. Furthermore, the resource scheduling and task migration processes of this solution lack a global coordination mechanism, making it prone to scheduling conflicts and resource contention in large-scale cluster environments, affecting the stability and predictability of the system.

[0009] To address these technical challenges, academia and industry have conducted extensive research. In adaptive resource scheduling, deep reinforcement learning-based scheduling algorithms adaptively adjust resource allocation strategies by learning patterns from historical data. However, these methods typically require large amounts of training data and have slow model convergence, making them ill-suited for rapidly changing workloads. Regarding incremental computation, incremental view maintenance and incremental machine learning algorithms can reuse historical computation results to reduce redundant computations. However, most existing methods are designed for specific types of computational tasks, lacking generality and scalability. As for fault tolerance mechanisms, while checkpoint-based fault tolerance methods can ensure the persistence of computational state, the frequency and granularity of checkpoints struggle to balance fault recovery time and system overhead. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a distributed big data real-time processing and analysis system. By introducing a multi-level elastic resource scheduling mechanism, fine-grained incremental state management technology and intelligent predictive fault tolerance strategy, it solves the problems of inflexible resource scheduling, low data processing efficiency and weak fault tolerance in existing technologies, and achieves efficient, low-latency and high-reliability processing of massive real-time data streams.

[0011] The technical solution adopted in this invention is as follows: A distributed big data real-time processing and analysis system is constructed. This system uses a multi-dimensional data stream acquisition module to extract features and classify real-time data streams from multiple data sources, generating data stream feature vectors to provide accurate input information for subsequent resource scheduling. The core elastic resource scheduling module employs an adaptive weight adjustment strategy, dynamically calculating the resource allocation weights of each computing node based on data stream characteristics and the current system state, achieving fine-grained scheduling and elastic scaling of computing resources. The incremental state management module, through a state snapshot mechanism and incremental computation graph technology, recalculates only the changed data fragments, significantly reducing computational overhead and improving processing efficiency. The intelligent fault-tolerant coordination module, through heartbeat detection and state snapshot recovery mechanisms, achieves rapid fault detection and task migration, ensuring high system availability. The real-time analysis output module aggregates and filters the calculation results, outputting analysis results according to user needs.

[0012] The beneficial effects of this invention are as follows: First, through a multi-level elastic resource scheduling mechanism, the system can dynamically adjust the memory capacity and number of processing threads of computing nodes according to the characteristics of real-time data streams and system load status, achieving fine-grained resource allocation and elastic scaling. Compared with traditional static resource allocation methods, the adaptive scheduling strategy of this invention can improve resource utilization by more than 40% and maintain stable processing performance under sudden traffic scenarios. Second, through fine-grained incremental state management technology, the system only recalculates changed data fragments, avoiding the overhead of full computation. Experimental results show that in typical window aggregation and join operation scenarios, incremental computation can reduce processing latency by more than 60% and increase throughput by more than 150%. Third, through an intelligent fault-tolerant coordination mechanism, the system can complete task migration and state recovery within 200ms after detecting a node failure, shortening the fault recovery time by 80% compared to traditional methods, and achieving system availability of 99.99%. Fourth, through the collaborative work of data stream feature vectors and state snapshot mechanisms, the system reduces checkpoint overhead by 50% while ensuring accurate one-time processing semantics, effectively improving the overall performance and reliability of the system. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the overall architecture of the distributed big data real-time processing and analysis system of the present invention.

[0014] Figure 2 This is a schematic diagram illustrating the workflow of the multidimensional data stream acquisition module and the elastic resource scheduling module of the present invention.

[0015] Figure 3 This is a schematic diagram of the incremental calculation process of the incremental state management module of the present invention.

[0016] Figure 4 This is a schematic diagram of the fault detection and recovery process of the intelligent fault-tolerant coordination module of the present invention.

[0017] Figure 5 This is a schematic diagram of the adaptive weight adjustment process of the elastic resource scheduling module of the present invention.

[0018] Figure 6 This is a schematic diagram of the state snapshot mechanism of the incremental state management module of the present invention.

[0019] Figure 7 This is a schematic diagram of the data processing flow of the real-time analysis output module of the present invention. Detailed Implementation

[0020] Please refer to the attached document. Figures 1-7To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0021] Reference Figure 1 The distributed big data real-time processing and analysis system provided by this invention includes a multi-dimensional data stream acquisition module 1, an elastic resource scheduling module 2, an incremental state management module 3, an intelligent fault-tolerant coordination module 4, and a real-time analysis output module 5. These five modules work together to form a highly efficient, low-latency, and highly reliable distributed real-time data processing system.

[0022] The multi-dimensional data stream acquisition module 1 is responsible for acquiring real-time data streams from multiple heterogeneous data sources, preprocessing the acquired data, and extracting features to generate data stream feature vectors. This module supports various data source accesses, including message queues, database change logs, IoT devices, and web crawlers. The acquired data streams first undergo format parsing and data cleaning to remove duplicate and invalid data. Then, the module extracts basic features such as timestamps, data types, data sizes, and data source identifiers, and classifies and labels the data streams according to business rules, indicating data priority and processing requirements. Preferably, the data stream feature vector includes 10-20 dimensions of features, each of which is standardized and mapped to a numerical range of 0-1. The module also weights and adjusts the feature vectors based on the arrival frequency of the data streams and historical processing records, enabling the feature vectors to more accurately reflect the real-time status and processing requirements of the data streams.

[0023] The elastic resource scheduling module 2 is the core scheduling engine of the system, responsible for dynamically adjusting the allocation of computing resources based on data flow characteristics and system status. This module receives data flow feature vectors from the multi-dimensional data flow acquisition module 1, and, combined with the current system resource usage status and task queue length, calculates the resource allocation weights for each computing node using an adaptive weight adjustment strategy. The module first calculates the CPU utilization, memory usage, and network bandwidth usage of each computing node within the current time window, and then calculates the comprehensive load index. When the comprehensive load index exceeds a preset load threshold, the module identifies high-load nodes and initiates a resource expansion process. Conversely, when the comprehensive load index is below the lower threshold, the module identifies low-load nodes and initiates a resource contraction process. In calculating the resource allocation weights, the module comprehensively considers the historical performance of nodes, current load status, and data flow priority, and uses a multi-objective optimization algorithm to determine the optimal resource allocation scheme. Preferably, the weight coefficient for high-priority data flows is 0.7-0.9, and the weight coefficient for low-priority data flows is 0.1-0.3. The elastic resource configuration scheme generated by the module includes the memory capacity, number of processing threads and network bandwidth quota that each compute node needs to increase or decrease. These configurations are dynamically applied to the compute nodes through the container orchestration system to achieve elastic scaling of resources.

[0024] In one embodiment of the present invention, the elastic resource scheduling module 2 employs an innovative adaptive weight adjustment algorithm, which achieves precise control over resource allocation by introducing dynamic weight factors. The core formula of this algorithm is:

[0025] ,

[0026] in, For the first Resource allocation weights for each computing node For the first Normalized value of the priority of the data stream processed by each node. For the first The normalized value of the current comprehensive load index of each node. For the first Normalized historical performance scores of each node As a priority weighting factor, As the load balancing factor, Historical performance factor This represents the total number of computational nodes in the system. In a preferred embodiment, The value is 0.5. The value is 0.3. The values ​​are set to 0.2, and these parameter values ​​have been verified through extensive experiments to achieve optimal resource utilization and system performance under various workloads. This formula achieves fair and efficient resource allocation by weighting a combination of priority, load, and historical performance, and by normalizing to ensure that the sum of the weights of all nodes is 1. When the data flow priority increases, the weight of the corresponding node increases, allocating more computing resources; when the node load increases, the weight decreases accordingly to avoid overload; nodes with good historical performance receive higher weights to fully utilize the processing power of high-performance nodes.

[0027] The incremental state management module 3 enables efficient incremental computing capabilities, a key technology for improving data processing efficiency. This module receives elastic resource configuration schemes from the elastic resource scheduling module 2 and performs sharding processing on the real-time data stream. Data shards are grouped according to timestamps and keys, with data of the same key value assigned to the same shard for state management. The module maintains a distributed state storage system to save the latest computational state of each data shard. When new data arrives, the module first extracts the historical state snapshot associated with that data, and identifies changed data records by comparing the differences between the current data and the historical state. Only these changed data records are processed; unchanged data directly reuses historical computation results. The module constructs a directed acyclic graph for incremental computing based on data dependencies and uses topological sorting to determine the execution order of computation tasks. During incremental computing, the module adopts a strategy to minimize computational cost, prioritizing tasks with small impact and low computational overhead. After each computation task is completed, the module immediately updates the corresponding state snapshot, generating a new state version. Preferably, the state snapshot adopts an incremental snapshot strategy, saving only state data that has changed compared to the previous snapshot, significantly reducing storage overhead. The state snapshot can be triggered in two ways: time interval triggering and event triggering. Time interval triggering is set to 1-5 seconds, while event triggering is executed when a critical computing node completes.

[0028] In one embodiment of the present invention, the incremental state management module 3 employs an innovative incremental computation cost evaluation algorithm to determine the optimal execution path for incremental computation. The core idea of ​​this algorithm is to quantify the incremental computation cost of each computation task and select the execution scheme with the minimum total cost. The formula for calculating the incremental computation cost is:

[0029] ,

[0030] in, For the task The incremental computational cost, For the task The set of data records that have changed involved The size of the set, For the task The computational complexity of a single record, For the task The set of predecessor tasks that depend on The data volume weighting coefficient. To calculate the complexity weighting coefficients, This depends on the propagation coefficient. In a preferred embodiment, The value is 0.4. The value is 0.5. The value is set to 0.1. This formula comprehensively considers the impact of three factors on the incremental computation cost: the amount of changing data, computational complexity, and task dependencies. By calculating the incremental computation cost of all tasks to be executed, the module can identify the execution path with the lowest cost and prioritize scheduling these tasks, thereby maximizing the efficiency of incremental computation. Experiments show that after adopting this algorithm, the average computation latency of the system is reduced by 60%, and the performance advantage of incremental computation is most significant in scenarios where the data change rate is 10%-30%.

[0031] The intelligent fault-tolerant coordination module 4 is responsible for monitoring the system's health status, enabling rapid fault detection and recovery. This module continuously monitors the operating status of each computing node through a heartbeat detection mechanism. Each computing node periodically sends a heartbeat message to the coordination node, containing the node identifier, timestamp, current load status, and information on currently executing tasks. The coordination node receives and records all heartbeat messages and calculates the heartbeat response time of each computing node. When a computing node's heartbeat response time exceeds a preset timeout threshold, or if no heartbeat message is received within three consecutive detection cycles, the module marks that node as a suspected faulty node. To avoid false positives, the module sends a probe request to the suspected faulty node for secondary confirmation. If there is still no response, it is confirmed as a faulty node, and the fault-tolerant recovery process is triggered. Preferably, the heartbeat detection cycle is set to 500ms-1000ms, and the timeout threshold is set to three times the detection cycle. After detecting a faulty node, the module immediately retrieves the list of tasks currently being executed by that node from the task allocation table and loads the latest state snapshots of these tasks from the state snapshot storage system. The state snapshot contains information such as task input data, intermediate computation results, and execution location, enabling a complete recovery of the task's execution context. Based on the elastic resource configuration scheme, the module selects nodes with sufficient resources and low load from healthy nodes as task migration targets. During task reallocation, the module prioritizes the rapid recovery of high-priority tasks, performing optimal matching based on task priority, resource requirements, and the available resources of the target node. After task reallocation is complete, the module updates the task allocation table and resource usage records, generates task reallocation instructions and resource adjustment instructions, and sends these instructions to the relevant computing nodes for execution.

[0032] In one embodiment of the present invention, the intelligent fault-tolerant coordination module 4 employs an innovative fault prediction algorithm to predict potential fault risks by analyzing the historical operating data of nodes, thereby achieving proactive fault tolerance. This algorithm uses a fault prediction model based on time series analysis, and its core formula is:

[0033] ,

[0034] in, For the first Each node in time The predicted failure probability value, The sigmoid activation function is used to map predicted values ​​to the range of 0-1. The number of feature dimensions. For the first One characteristic in time The observed values, For the first The weight parameters corresponding to each feature. In a preferred embodiment, Set to 8, features include CPU utilization variance, memory usage growth rate, network latency jitter, disk I / O latency, task failure rate, heartbeat response time standard deviation, error log frequency, and system load peak frequency. Weighting parameters. The data is obtained through logistic regression training on historical failure event data. Preferably, the weight of CPU utilization fluctuation variance is 0.25, the weight of memory usage growth rate is 0.20, and the weights of other features are between 0.05 and 0.15. When the failure risk exceeds the preset threshold of 0.8, the module triggers task migration in advance, proactively transferring some tasks of the affected node to other healthy nodes to avoid passive recovery in the event of a failure. Experiments show that this failure prediction algorithm can predict node failures 10-30 seconds in advance, further reducing the system's failure recovery time by 50%, and significantly improving the system's availability and stability.

[0035] The real-time analysis output module 5 is responsible for the final aggregation, filtering, and output of the calculation results. This module receives the fault-tolerance recovery strategy from the intelligent fault-tolerance coordination module 4 and the incremental calculation result set from the incremental state management module 3, collecting and summarizing the distributed calculation results output by each computing node. First, the module groups the calculation results according to the data's key values, aggregating results with the same key values ​​together. Then, the module performs aggregation operations on the grouped results, supporting aggregation functions including summation, average, maximum value, minimum value, count, and quantiles. The module filters the aggregation results according to user-defined query conditions, supporting various methods such as time range filtering, numerical range filtering, and regular expression matching. The filtered analysis results are sorted chronologically, and the module generates output data according to the user-specified output format. Supported output formats include JSON, CSV, Avro, and database record formats. The generated output data can be sent to an external system for real-time consumption via a message queue or written to a distributed storage system for persistent storage for offline analysis. Preferably, the module uses an asynchronous write mechanism, writing the output data to a local buffer first, and then writing it in batches to the target system, avoiding performance bottlenecks caused by synchronous writes. The module also records the timestamp and status information of each output operation for auditing and troubleshooting.

[0036] Reference Figure 2The workflow of the multi-dimensional data stream acquisition module 1 and the elastic resource scheduling module 2 includes four main stages: data acquisition, feature extraction, load monitoring, and resource scheduling. In the data acquisition stage, the module collects real-time data streams in parallel from multiple data sources, including Kafka message queues, database change streams, and IoT gateways. The collected raw data streams are first parsed, unifying data from different formats such as JSON, XML, and Avro into a unified internal data structure. Then, data cleaning is performed to remove duplicate data, filter invalid data, and correct format errors. In the feature extraction stage, the module extracts features such as timestamps, data types, data sizes, data source identifiers, business tags, and priorities to construct multi-dimensional feature vectors. The feature vectors are normalized and mapped to the 0-1 range, and weighted according to the arrival frequency of the data stream and historical processing records. In the load monitoring stage, the elastic resource scheduling module 2 continuously monitors the resource usage of each computing node, including CPU utilization, memory usage, network bandwidth utilization, disk I / O, and task queue length. The module calculates the comprehensive load index of each node and compares it with a preset load threshold to identify high-load and low-load nodes. In the resource scheduling phase, the module calculates the resource allocation weights for each node using an adaptive weight adjustment algorithm based on data flow feature vectors and node load status. The module generates elastic resource configuration schemes, including the amount of memory capacity, number of processing threads, and network bandwidth quotas that need to be increased or decreased. These configurations are dynamically applied to compute nodes through container orchestration systems such as Kubernetes or Docker Swarm, enabling automatic resource scaling.

[0037] Reference Figure 3The incremental computation process of Incremental State Management Module 3 includes five steps: data sharding, state comparison, incremental identification, computation graph construction, and incremental execution. In the data sharding step, the module receives real-time data streams and groups and shards the data according to its timestamp and key value. Data with the same key value are assigned to the same shard, facilitating subsequent state management and incremental computation. The sharding strategy uses a consistent hashing algorithm to ensure uniform distribution and load balancing of data shards. In the state comparison step, the module loads historical state snapshots corresponding to each data shard from the distributed state storage system. State snapshots are versioned, with each snapshot containing a version number, timestamp, and state data. The module compares the current data shard with each historical state snapshot, identifying changed data records through hash value comparison or data content comparison. In the incremental identification step, the module marks changed data records as pending data, while unchanged data directly reuses historical computation results. For newly added data, the module adds it to the pending data set; for deleted data, the module removes the corresponding record from the state snapshot; for updated data, the module updates the corresponding record in the state snapshot. In the computation graph construction step, the module constructs a directed acyclic graph (DAG) for incremental computation based on predefined computation logic and data dependencies. Each node in the graph represents a computation task, and edges represent dependencies between tasks. The module uses a topological sorting algorithm to determine the execution order of computation tasks, ensuring that predecessor tasks are completed before successor tasks. In the incremental execution step, the module executes incremental computation tasks sequentially according to the topological order of the computation graph. After each task is completed, the module immediately updates the corresponding state snapshot, generating a new state version. After all tasks are completed, the module summarizes and outputs the incremental computation results.

[0038] Reference Figure 4The fault detection and recovery process of the intelligent fault-tolerant coordination module 4 includes five stages: heartbeat monitoring, fault identification, state loading, task reallocation, and recovery confirmation. In the heartbeat monitoring stage, each computing node periodically sends heartbeat messages to the coordination node. The heartbeat message includes the node identifier, timestamp, CPU utilization, memory usage, network latency, a list of currently executing tasks, and task execution progress. After receiving the heartbeat message, the coordination node records the last heartbeat time and node status information for each node. In the fault identification stage, the coordination node continuously calculates the heartbeat response time of each node. When the heartbeat response time of a node exceeds a preset timeout threshold, or if no heartbeat message is received within three consecutive detection cycles, the coordination node marks that node as a suspected faulty node. To avoid misjudgments caused by network jitter, the coordination node sends a probe request to the suspected faulty node for secondary confirmation. If no response is received within 2 seconds, the node is confirmed as a faulty node, and the fault-tolerant recovery process is triggered. In the state loading stage, the coordination node retrieves the list of tasks currently being executed by the faulty node from the task allocation table and loads the latest state snapshots of these tasks from the distributed state snapshot storage system. The state snapshot contains complete information such as the task's input data, intermediate calculation results, execution location, and the status of dependent tasks. During the task reassignment phase, the coordinating node selects a node with sufficient resources and low load from healthy nodes as the task migration target based on the current elastic resource configuration scheme. The module uses a greedy algorithm for task node matching, prioritizing the rapid recovery of high-priority tasks. During task reassignment, the module considers the task's resource requirements and the available resources of the target node, ensuring that the target node has sufficient CPU, memory, and network resources to execute the migrated task. In the recovery confirmation phase, the coordinating node sends the task reassignment command and state snapshot to the target node. The target node loads the state snapshot, restores the task's execution context, and continues execution from the point of interruption. After successful task recovery, the target node sends a confirmation message to the coordinating node, which then updates the task allocation table and resource usage records.

[0039] Reference Figure 5The adaptive weight adjustment process of the elastic resource scheduling module 2 includes five steps: feature extraction, load calculation, performance evaluation, weight calculation, and resource adjustment. In the feature extraction step, the module receives data stream feature vectors from the multi-dimensional data stream acquisition module 1 and extracts features such as arrival rate, packet size, processing complexity, and priority. Simultaneously, the module obtains real-time resource usage data for each computing node from the system monitoring module, including CPU utilization, memory usage, network bandwidth usage, disk I / O, and task queue length. In the load calculation step, the module normalizes the resource usage data of each node, mapping indicators of different dimensions to the 0-1 range. The module assigns weight coefficients to CPU utilization, memory usage, and network bandwidth usage; preferably, the weight for CPU utilization is 0.4, the weight for memory usage is 0.35, and the weight for network bandwidth usage is 0.25. The module then performs a weighted summation of the normalized resource utilization rates with their corresponding weight coefficients to generate a comprehensive load index for each node. In the performance evaluation step, the module extracts the operational data of each node over a historical time period and calculates performance metrics such as average response time, task completion rate, and throughput. Based on these metrics, the module calculates a historical performance score for each node, reflecting its processing capacity and stability. In the weight calculation step, the module employs an adaptive weight adjustment algorithm, comprehensively considering three dimensions—data flow priority, node load, and historical performance—to calculate the resource allocation weight for each node. The parameters in the weight calculation formula are... , and Dynamically adjust based on the current system workload type, increasing the load when high-priority tasks account for a large proportion. Increase when the load is unbalanced Increase when there are large differences in node performance. In the resource adjustment step, the module determines the amount of computing resources that each node needs to increase or decrease based on the calculated resource allocation weights and the current task queue length. The module generates resource adjustment instructions, including adjustments to memory capacity, the number of processing threads, and network bandwidth quotas. These instructions are issued to each compute node through the container orchestration system's API interface, enabling dynamic scaling of resources.

[0040] Reference Figure 6The incremental state management module 3's state snapshot mechanism comprises five stages: snapshot triggering, state serialization, incremental compression, snapshot storage, and state recovery. In the snapshot triggering stage, the module initiates state snapshot operations based on preset trigger conditions. These conditions include timed triggering and event triggering. Timed triggering executes snapshots at fixed time intervals, dynamically adjusted based on the data flow arrival rate and system load, with an optimal range of 1-5 seconds. Event triggering executes snapshots upon completion of critical computation nodes, such as window closing, aggregation completion, and state updates. In the state serialization stage, the module serializes the current computational state into a byte stream. Computational states include window aggregation results, join state tables, intermediate results of iterative computations, and user-defined states. The module employs efficient serialization frameworks such as Kryo or Protobuf for state serialization, achieving a 3-5 times performance improvement compared to Java's native serialization. In the incremental compression stage, the module uses an incremental snapshot strategy, saving only state data that has changed compared to the previous snapshot. The module first loads the previous snapshot, identifies changed state entries through hash value comparison, and packages these changed entries into an incremental snapshot. For the first snapshot or a full snapshot, the module saves complete state data. Incremental snapshot data is compressed using algorithms such as LZ4 or Snappy to further reduce storage space usage. During snapshot storage, the module stores the compressed snapshot data in a distributed storage system such as HDFS or object storage. The module assigns a unique version number and timestamp to each snapshot; the version number is a monotonically increasing integer, and the timestamp records the precise time the snapshot was created. The module builds a snapshot index, saving snapshot metadata, including version number, timestamp, task identifier, and storage path, to an index table for easy retrieval. In the state recovery phase, when task state needs to be restored, the module searches the snapshot index for the latest available snapshot based on the task identifier and recovery time point. The module loads the snapshot data from the distributed storage system, decompresses and deserializes the compressed snapshot data, and restores the task's execution state. For incremental snapshots, the module needs to load the base snapshot and all incremental snapshots, applying incremental changes sequentially to reconstruct the complete state.

[0041] Reference Figure 7The real-time analysis output module 5's data processing flow includes five steps: result collection, data grouping, aggregation calculation, result filtering, and data output. In the result collection step, the module receives the output results of incremental calculations from each computing node. The output of each computing node includes the data's key value, the calculated result value, and a timestamp. The module collects the distributed calculation results to a central node for unified processing. To avoid network transmission becoming a bottleneck, the module employs data compression and batch transmission technologies, packaging multiple result records into a single transmission batch to reduce network overhead. In the data grouping step, the module groups the result data according to its key value, aggregating results with the same key value together. The grouping operation uses a hash partitioning algorithm to ensure that data with the same key value is assigned to the same processing partition. The module supports multi-level grouping, allowing hierarchical grouping based on primary key, secondary key, and time window. In the aggregation calculation step, the module executes user-defined aggregation functions on the grouped result data. Supported aggregation functions include statistical functions such as summation, average, maximum, minimum, count, standard deviation, variance, and quantiles. The module uses an incremental aggregation algorithm, reusing historical aggregation results and performing incremental aggregation only on newly added data, improving computational efficiency. In the result filtering step, the module filters the aggregation results based on user-defined query conditions. Query conditions support time range filtering, numerical range filtering, regular expression matching, and user-defined filtering functions. The module employs predicate pushdown technology, applying filtering conditions before aggregation calculation to reduce the amount of data processed subsequently. Filtered results are sorted by timestamp or numerical value, supporting ascending and descending order. In the data output step, the module generates the final output data according to the user-specified output format and output target. For real-time output scenarios, the module sends the result data to message queues such as Kafka and RabbitMQ for downstream systems to consume in real time. For persistent storage scenarios, the module writes the result data to HDFS, HBase, or relational databases for offline analysis. The module uses asynchronous and batch write strategies, first writing output data to a local buffer, and then batch writing it to the target system when the buffer reaches a certain size or after a certain time. The module records the timestamp, output data volume, and operation status of each output operation for auditing and monitoring.

[0042] In practical applications, the distributed big data real-time processing and analysis system of this invention has been successfully deployed in multiple fields such as financial risk control, telecommunications network monitoring, and IoT data analysis. Taking the financial risk control scenario as an example, the system needs to perform real-time risk assessment on millions of transaction data per second to identify abnormal transactions and fraudulent activities. The system is deployed on a cluster of 100 computing nodes, each configured with a 16-core CPU, 64GB of memory, and 1Gbps of network bandwidth. Through the adaptive scheduling strategy of the elastic resource scheduling module, the system can dynamically adjust computing resources according to fluctuations in transaction traffic, automatically expanding to 150 nodes during peak trading hours and shrinking to 50 nodes during off-peak hours, improving resource utilization by 45% and saving costs by 35%. Through the incremental computing technology of the incremental state management module, the system achieves a 70% reuse rate for historical transaction data, recalculating only 30% of the changed data, reducing processing latency from 500ms for traditional full-data computation to 180ms, meeting the stringent real-time requirements of financial operations. Through the fault prediction and rapid recovery mechanism of the intelligent fault-tolerant coordination module, the system experienced 23 node failures in the past year of operation, with an average fault recovery time of 150ms, and the system availability reached 99.995%, with no data loss events.

[0043] In telecommunications network monitoring scenarios, the system needs to monitor network traffic and signaling data from millions of users across the network in real time, identifying network anomalies and service quality issues. The system processes 8 million data events per second, with a data throughput of 12GB / s. Through the multi-dimensional data stream acquisition module's classification and labeling of different data stream types, the system can allocate higher processing priority to critical signaling data, ensuring the real-time performance of core services. Through the multi-objective optimization algorithm of the elastic resource scheduling module, the system optimizes resource costs while maintaining processing latency, achieving a 40% cost saving compared to fixed resource configuration schemes. Through the fine-grained state management of the incremental state management module, the system can efficiently maintain the session state and statistical information of each user, with state update latency controlled within 100ms. The real-time analysis output module pushes analysis results to the network operation and maintenance platform in real time, enabling maintenance personnel to discover and handle network problems immediately. The average network fault detection time has been reduced from the traditional 5 minutes to 30 seconds, significantly improving network service quality.

[0044] In IoT data analytics scenarios, the system needs to process and analyze massive amounts of sensor data from smart manufacturing, smart cities, and connected vehicles in real time. The system connects to over 10 million IoT devices, generating 50 million data events per second. Through the multi-source data integration capabilities of the multi-dimensional data stream acquisition module, the system can uniformly collect and process data from devices from different manufacturers and using different protocols. Through the dynamic scaling capabilities of the elastic resource scheduling module, the system can cope with the suddenness and uncertainty of IoT data, automatically expanding computing resources when devices report large amounts of data and shrinking resources when data is sparse, achieving elastic and economical resource utilization. Through the state snapshot mechanism of the incremental state management module, the system can support long-term tracking and historical review of device status, facilitating fault diagnosis and predictive maintenance. Through the high availability guarantee of the intelligent fault-tolerant coordination module, the system maintains extremely high stability during 24 / 7 continuous operation, providing reliable data processing capabilities for smart manufacturing and smart city applications.

[0045] The key technical indicators of system performance are as follows: In terms of throughput, a single node can process 100,000 events / second, a 100-node cluster can reach 10 million events / second, and a large cluster of 1000 nodes can reach 100 million events / second. Regarding latency, the end-to-end processing latency P99 percentile is 200ms, the average latency is 80ms, and the lowest latency under low load scenarios can reach 30ms. In terms of resource utilization, CPU utilization reaches 75%-85% under high load, memory utilization reaches 70%-80%, and network bandwidth utilization reaches 60%-70%, representing an overall resource utilization improvement of 40%-50% compared to traditional fixed resource configuration schemes. In terms of availability, the system availability reaches over 99.99%, the mean time to recovery (MTTR) is 150ms, and the mean time between failures (MTBF) exceeds 30 days. In terms of scalability, the system supports linear scaling from 10 nodes to 1000 nodes without downtime, and new nodes can join the cluster and begin processing data within 5 minutes.

[0046] The above description is only a preferred embodiment of the present invention and does not limit the scope of protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection of the present invention.

Claims

1. A distributed big data real-time processing and analysis system, characterized in that, include: The multi-dimensional data stream acquisition module is used to acquire real-time data streams from multiple data sources, perform feature extraction and classification labeling on the acquired data streams, generate data stream feature vectors based on the data timestamps, data types and priorities, and output the data stream feature vectors to the elastic resource scheduling module. The elastic resource scheduling module is connected to the multi-dimensional data stream acquisition module. Based on the data stream feature vector, it analyzes the current system resource usage status and task queue length. Combined with historical resource allocation records, it uses an adaptive weight adjustment strategy to calculate the resource allocation weight of each computing node. Based on the resource allocation weight, it dynamically adjusts the memory capacity and number of processing threads of the computing nodes to generate an elastic resource configuration scheme. The elastic resource configuration scheme is then output to the incremental status management module. The specific process for generating the elastic resource allocation scheme is as follows: Based on the data flow feature vector, the arrival rate, data packet size, and processing complexity parameters of the data flow are extracted; the CPU utilization, memory usage, and network bandwidth utilization of each computing node within the current time window are statistically analyzed; based on the CPU utilization, memory usage, and network bandwidth utilization, the comprehensive load index of each computing node is calculated; the comprehensive load index is compared with a preset load threshold to identify high-load and low-load nodes; based on the identification results of high-load and low-load nodes, combined with the resource usage patterns in historical resource allocation records, an adaptive weight adjustment strategy is used to calculate the resource allocation weight of each computing node; based on the resource allocation weight and the current task queue length, the amount of computing resources that need to be increased or decreased is determined, a resource adjustment instruction is generated, and the memory capacity and number of processing threads of the computing nodes are dynamically adjusted according to the resource adjustment instruction to generate an elastic resource allocation scheme; The calculation process for the resource allocation weights is as follows: Resource usage data for each computing node over a historical time period is extracted, including average CPU utilization, average memory usage, and average network bandwidth usage; the average response time and task failure rate for each computing node are calculated; based on the average response time and task failure rate, a performance score for each computing node is calculated; the performance score is weighted and combined with the comprehensive load index to generate a comprehensive node evaluation value; based on the comprehensive node evaluation value, a normalization method is used to map the evaluation value of each computing node to a unified weight range; the normalized comprehensive node evaluation value is dynamically adjusted according to the data flow priority and task type; and the resource allocation weights for each computing node are calculated using the following adaptive weight adjustment algorithm formula: in, For the first Resource allocation weights for each computing node For the first Normalized value of the priority of the data stream processed by each node. For the first The normalized value of the current comprehensive load index of each node. For the first Normalized historical performance scores of each node As a priority weighting factor, As the load balancing factor, Historical performance factor This represents the total number of computing nodes in the system. The incremental state management module is connected to the elastic resource scheduling module. Based on the elastic resource configuration scheme, it performs incremental calculations on the real-time data stream, records the intermediate states during the calculation process through a state snapshot mechanism, and recalculates only the changed data fragments when a state change is detected. It constructs an incremental calculation graph based on data dependencies, determines the incremental calculation path using a strategy to minimize the calculation cost, generates an incremental calculation result set, and outputs the incremental calculation result set to the intelligent fault-tolerant coordination module. The implementation process of the state snapshot mechanism is as follows: During incremental computation, a state snapshot operation is triggered periodically or when a key computation node is completed, serializing the current computation state into state snapshot data; the state snapshot data is compressed, and an incremental snapshot strategy is adopted to save only the state snapshot data that has changed compared to the previous snapshot, generating compressed incremental snapshots; the compressed incremental snapshots are stored in a distributed storage system, and each snapshot is assigned a unique version number and timestamp, and a snapshot index is established for quick retrieval; when it is necessary to restore the task state, the latest state snapshot is found from the snapshot index according to the task identifier and timestamp, the state snapshot is loaded from the distributed storage system, the snapshot data is decompressed and deserialized, and the execution state of the task is restored; The intelligent fault-tolerant coordination module is connected to the incremental state management module. Based on the incremental calculation result set, it monitors the health status and task execution progress of each computing node, identifies faulty nodes through a heartbeat detection mechanism, and restores the execution status of the faulty task from the state snapshot when a node fault is detected. The faulty task is then reassigned to a healthy node, the task scheduling order is adjusted according to task priority and resource availability, a fault-tolerant recovery strategy is generated, and the fault-tolerant recovery strategy is output to the real-time analysis output module. The heartbeat detection mechanism is implemented as follows: each computing node periodically sends a heartbeat message to the coordination node. The heartbeat message includes a node identifier, timestamp, current load status, and information on the tasks being executed. The coordination node receives and records the heartbeat messages from each computing node, calculates the heartbeat response time of each computing node, and compares the heartbeat response time with a preset timeout threshold. When the heartbeat response time of a computing node exceeds the timeout threshold, or when no heartbeat message is received within three consecutive detection cycles, the computing node is marked as a suspected faulty node. A probe request is sent to the suspected faulty node for secondary confirmation. If there is still no response, it is confirmed as a faulty node, triggering a fault tolerance recovery process. The task information of the faulty node is obtained from the task allocation table, and task migration and state recovery operations are initiated. The intelligent fault-tolerant coordination module also employs a fault prediction algorithm to predict potential fault risks by analyzing the historical operating data of nodes, thereby achieving proactive fault tolerance. The formula for the fault prediction algorithm is as follows: in, For the first Each node in time The predicted failure probability value, The Sigmoid activation function is used to map predicted values ​​to the range of 0-1. The number of feature dimensions. For the first One characteristic in time The observed values, For the first The weight parameters corresponding to each feature, when When the preset fault risk threshold is exceeded, task migration is triggered in advance, and some tasks of the node are actively transferred to other healthy nodes. The real-time analysis output module is connected to the intelligent fault-tolerant coordination module. Based on the fault-tolerant recovery strategy and the incremental calculation result set, it aggregates and summarizes the calculation results, filters target data according to user-defined query conditions, and outputs the filtered analysis results to an external system or stores them in a distributed storage system in a specified format.

2. The distributed big data real-time processing and analysis system according to claim 1, characterized in that, The specific process for generating the incremental calculation result set is as follows: Based on the aforementioned elastic resource allocation scheme, the real-time data stream is segmented, and the data segments are grouped according to the timestamp and key value to generate a data segment set. Traverse the data shard set, extract the associated historical state snapshot for each data shard, and identify the changed data records by comparing the differences between the current data shard and the historical state snapshot; Only the data records that have changed are processed. Based on the predefined calculation logic and data dependencies, a directed acyclic graph for incremental calculation is constructed to determine the execution order of incremental calculation. The incremental computation tasks are executed sequentially according to the execution order. The output results of each computation task are stored in a state snapshot to generate a new state version. The output results of all incremental computation tasks are summarized to generate an incremental computation result set.

3. The distributed big data real-time processing and analysis system according to claim 2, characterized in that, The specific process for generating the fault-tolerant recovery strategy is as follows: Based on the incremental calculation result set, the running status of each computing node is monitored through a heartbeat detection mechanism, and the heartbeat response time and task execution progress of each computing node are recorded. When a computing node is detected to have a heartbeat response timeout or an abnormal task execution, the node is marked as a faulty node, and the list of tasks being executed by the faulty node is retrieved from the task allocation table. Load the latest state snapshot of each task in the task list from the state snapshot storage system, and restore the execution context of the task based on the state snapshot, including input data, intermediate results and execution location; Based on the aforementioned elastic resource allocation scheme, nodes with sufficient resources and low load are selected from healthy nodes. According to task priority and resource availability, the tasks of the faulty nodes are reallocated to the selected healthy nodes. Update the task allocation table and resource usage records, generate task reassignment instructions and resource adjustment instructions, and combine the task reassignment instructions and resource adjustment instructions to generate a fault-tolerant recovery strategy.

4. The distributed big data real-time processing and analysis system according to claim 3, characterized in that, The specific process for generating the data stream feature vector is as follows: Real-time data streams are collected from multiple data sources, and the collected data streams are parsed and cleaned to remove duplicate and invalid data. Feature extraction is performed on the cleaned data stream, including the timestamp, data type, data size, and data source identifier. The data stream is then classified and labeled according to business rules, with the data priority and processing requirements marked. Based on the timestamp, data type, data size, data source identifier, and priority, a multidimensional feature vector is constructed. The multidimensional feature vector is then standardized to map each feature value to a uniform numerical range. Based on the arrival frequency of the data stream and historical processing records, the multidimensional feature vector is weighted and adjusted to generate a data stream feature vector.

5. The distributed big data real-time processing and analysis system according to claim 4, characterized in that, The processing procedure of the real-time analysis output module is as follows: Based on the fault-tolerant recovery strategy and the incremental calculation result set, the calculation results output by each computing node are collected and summarized, and the calculation results are grouped according to the key value of the data. Aggregate the grouped calculation results, including summation, average, maximum value, minimum value, and count, and filter the aggregated results according to user-defined query conditions; The filtered aggregation results are sorted in chronological order, and output data is generated according to the output format specified by the user, including JSON format, CSV format and database record format; The generated output data is sent to an external system via a message queue, or written to a distributed storage system for persistent storage, recording the timestamp and status information of the output operation.

6. The distributed big data real-time processing and analysis system according to claim 1, characterized in that, The calculation process for the comprehensive load index is as follows: The CPU utilization, memory usage, and network bandwidth utilization of each computing node within the current time window are statistically analyzed, and the CPU utilization, memory usage, and network bandwidth utilization are normalized. Based on the importance of different resource types, weight coefficients are assigned to CPU utilization, memory usage, and network bandwidth utilization. The normalized utilization of each resource is then weighted and summed with its corresponding weight coefficient to generate a comprehensive load index.

Images