Large data real-time processing and distributed storage optimization software system supporting high concurrency

Abstract

The invention discloses a big data real-time processing and distributed storage optimization software system supporting high concurrency, which is characterized in that a multi-level data access gateway is constructed, multi-source heterogeneous concurrent data flow is preprocessed, and the preprocessing comprises dynamic flow limiting based on flow characteristics, data format standardization and redundant data filtering; wherein the dynamic flow limiting adopts a mode of combining a token bucket algorithm and data priority weight distribution, core service data flow is released preferentially, and when the concurrent flow exceeds a preset threshold by 10-100 times, an elastic buffer queue mechanism is triggered to avoid data loss; through the technical innovation of the whole process of access, processing, storage and monitoring, a set of complete high-concurrency big data real-time processing and distributed storage optimization scheme is formed, and the method can be widely applied to the fields of finance, e-commerce, logistics and the like with high requirements for concurrency capability, real-time performance and data security, and has extremely high practicability and popularization value.

Description

Technical Field

[0001] This invention relates to data processing, and more particularly to a software system that supports high-concurrency real-time processing and distributed storage optimization for big data. Background Technology

[0002] With the deep integration of big data technology and the digital economy, business data flows in fields such as finance, e-commerce, and logistics exhibit significant characteristics of "high concurrency, heterogeneity, and strong real-time requirements." Financial transaction scenarios, in particular, require processing tens to hundreds of thousands of transactions per second, placing stringent demands on data processing latency, accuracy, and storage security. While real-time big data processing and distributed storage systems are now widely used, numerous technical bottlenecks remain in high-concurrency scenarios, making it difficult to meet the actual needs of core businesses.

[0003] At the high-concurrency data access level, traditional data access gateways mostly adopt fixed threshold rate limiting mechanisms, lacking differentiated scheduling of data priorities. Core business data streams and non-core data streams compete for resources, which can easily lead to excessive latency in core businesses. At the same time, when faced with sudden traffic peaks, the buffer queue capacity is fixed and cannot flexibly adapt to traffic fluctuations, often resulting in data loss or access blockage. Furthermore, the standardization and cleaning efficiency of multi-source heterogeneous data is low, making it difficult to support the parallel processing of millions of data points per second.

[0004] At the data processing level, existing technologies mostly adopt simple stream processing or batch processing architectures. While stream processing can guarantee real-time performance, it suffers from insufficient data processing accuracy, weak fault recovery capabilities, and is prone to data duplication or loss. While batch processing can guarantee data accuracy, it has high latency and cannot meet real-time business needs. Some integrated stream and batch architectures have fixed micro-batch durations, which cannot be dynamically adjusted according to throughput and latency requirements, making it difficult to balance real-time performance and processing efficiency. Furthermore, they lack a robust data lineage tracing mechanism, requiring full reprocessing after a failure, resulting in long recovery times.

[0005] At the distributed storage level, traditional storage systems mostly adopt static data allocation strategies, without dynamically scheduling storage resources based on data popularity, access frequency, and node load status. Centralized storage of hot data leads to node overload, while cold data occupies high-performance storage resources, resulting in waste. At the same time, the data sharding and replica management mechanisms are rigid, resulting in low efficiency in processing ultra-large data and the accumulation of storage fragments affecting data read speed, making it impossible to achieve the optimal balance between storage cost and access efficiency.

[0006] At the system operation and maintenance optimization level, the existing monitoring system focuses on the collection of single indicators and lacks linkage analysis of multi-dimensional indicators such as processing latency, node load, and storage IO, making it impossible to predict system bottlenecks in a timely manner; resource scheduling and storage optimization rely heavily on manual intervention, resulting in delayed response and difficulty in adapting to dynamic load changes in high-concurrency scenarios, leading to low overall system operating efficiency and insufficient stability.

[0007] Therefore, developing a big data processing and storage system that can adapt to high concurrency traffic, balance real-time performance and accuracy, optimize storage resource allocation, and has dynamic self-optimization capabilities has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] In order to overcome the shortcomings of the prior art, one of the objectives of this invention is to provide a software system that supports high-concurrency real-time processing of big data and distributed storage optimization.

[0009] One of the objectives of this invention is achieved through the following technical solution: A software system that supports high-concurrency real-time big data processing and distributed storage optimization includes the following steps: S1: Construct a multi-level data access gateway to preprocess multi-source heterogeneous concurrent data streams. The preprocessing includes dynamic rate limiting based on traffic characteristics, data format standardization, and redundant data filtering. The dynamic rate limiting adopts a combination of token bucket algorithm and data priority weight allocation to prioritize the passage of core business data streams. When the concurrent traffic exceeds the preset threshold by 10-100 times, an elastic buffer queue mechanism is triggered to avoid data loss. S2: Adopts a stream-batch integrated processing architecture. Through an improved micro-batch processing engine, the continuous data stream is divided into micro-batch units with dynamically adjustable durations. The duration of the micro-batch is adaptively adjusted within the range of 0.1-10 seconds according to the real-time throughput and latency requirements. At the same time, a data lineage tracking mechanism is introduced to achieve accurate one-time semantic processing of data after a fault, ensuring that the data is not duplicated or lost. S3: Distributed storage scheduling is performed based on data characteristics and node load status. The key-value distribution characteristics, hot attributes and access frequency of the processed data are extracted. The optimal storage node is allocated to the data through a two-layer grouping optimization algorithm. The dynamic mapping relationship between data and storage nodes is maintained. Hot data is stored with multiple replicas. The number of replicas is dynamically adjusted to 2-5 according to the access frequency. Cold data is automatically migrated to low-cost storage nodes. S4: Establish a closed loop of real-time monitoring and dynamic optimization, collect data processing latency, node load rate, storage IO rate and other indicators in real time. When node load imbalance or processing latency exceeds the preset threshold is detected, task migration and resource expansion are automatically triggered. At the same time, periodic defragmentation and index optimization of storage data are performed to improve data reading efficiency.

[0010] The multi-level data access gateway mentioned in step S1 includes an edge access layer, a traffic control layer, and a data preprocessing layer. The edge access layer adopts a distributed deployment mode and supports multi-message middleware protocol adaptation such as Kafka and RabbitMQ. The traffic control layer dynamically adjusts the token bucket issuance rate by real-time statistics of the peak traffic and fluctuation coefficient of each data source. The data preprocessing layer adopts a parallel parsing algorithm to achieve format conversion and cleaning of millions of data points per second.

[0011] The improved micro-batch processing engine described in step S2 is based on Spark Streaming optimization and introduces an adaptive batch size adjustment mechanism. It predicts changes in subsequent data stream throughput through a predictive model and adjusts the micro-batch unit size in advance. At the same time, it combines the RDD lineage mechanism and distributed checkpoint technology to control the fault recovery time within seconds and the core business data processing latency does not exceed 500 milliseconds.

[0012] The two-layer grouping optimization algorithm described in step S3 includes two stages: global node screening and local optimal allocation. In the global node screening stage, candidate storage nodes are selected based on the remaining storage capacity of the nodes, CPU utilization, and network bandwidth. In the local optimal allocation stage, the optimal storage scheme is generated with data transmission cost, storage cost, and access latency as objective functions. At the same time, a data sharding strategy is adopted to divide the ultra-large data into multiple shards by key-value hash to achieve parallel storage and retrieval.

[0013] The real-time monitoring system described in step S4 adopts a Prometheus+Grafana architecture, focusing on monitoring the number of asynchronous partitions in UnderReplicatedPartitions, the maximum consumption latency in RecordsLagMax, and the storage IO idle rate. Resource expansion adopts a containerized elastic scaling mode, supporting a combination of automatic expansion based on preset rules and manual intervention. Defragmentation is performed in the background using a low-priority thread to avoid affecting real-time data processing performance.

[0014] include: The data access optimization module is used to build a multi-level data access gateway to achieve dynamic rate limiting, format standardization and redundant data filtering for multi-source heterogeneous concurrent data streams, adapt to multiple message middleware protocols, and ensure stable data access in high-concurrency scenarios. The integrated stream and batch processing module is used to achieve adaptive micro-batch partitioning and processing of data streams through an improved micro-batch processing engine. Combined with data lineage tracking and fault recovery mechanisms, it ensures the accuracy and real-time performance of data processing and supports unified processing and integration of stream and batch data. The distributed storage scheduling module is used to allocate storage nodes based on data characteristics and node load status through a two-layer grouping optimization algorithm, maintain the dynamic mapping relationship between data and nodes, realize multi-copy storage of hot data and tiered storage of cold data, and optimize storage resource utilization. The monitoring and optimization module is used to collect system operation indicators in real time, trigger dynamic resource scheduling and storage optimization operations, perform data fragmentation defragmentation, index optimization and fault warning, and ensure the continuous and efficient operation of the system.

[0015] The data access optimization module has a built-in traffic prediction unit that analyzes historical traffic data through machine learning algorithms, predicts the arrival time of traffic peaks, and adjusts the rate limiting strategy and buffer queue capacity in advance. The integrated stream and batch processing module supports seamless integration with SparkSQL and MLlib components to achieve collaborative execution of real-time data analysis and machine learning tasks.

[0016] Furthermore, it includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for supporting high-concurrency real-time big data processing and distributed storage optimization as described in any one of claims 1 to 5.

[0017] Furthermore, the computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method for supporting high-concurrency real-time big data processing and distributed storage optimization as described in any one of claims 1 to 5.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention, through a multi-level data access gateway and dynamic rate limiting buffer mechanism, combined with a data priority weight allocation strategy, achieves differentiated scheduling of different business data streams. Compared with traditional fixed threshold rate limiting methods, it can adapt to 100,000 to 500,000 concurrent traffic per second, with no data loss under sudden traffic peaks. Core business data streams are given priority access, and latency is controlled within 500ms, fully meeting the stringent requirements of core scenarios such as financial transactions. The multi-regional distributed deployment design of the edge access layer further reduces data access latency, improves system fault tolerance in high-concurrency scenarios, and avoids the impact of single-region node failure on the overall access service.

[0019] This embodiment solves the problems of data loss, high latency, and waste of storage resources under high concurrency in traditional big data systems through the collaborative design of multi-level access gateways, an improved integrated stream and batch processing engine, two-layer group storage scheduling, and dynamic monitoring optimization. It achieves three core advantages: ① High concurrency adaptability, supporting stable access of 500,000 data streams per second with no data loss under peak traffic; ② Highly efficient real-time processing, with core business latency controlled within 500ms and fault recovery time in seconds; ③ Optimized and efficient storage, reducing storage costs and improving data access efficiency through tiered storage, defragmentation, and index optimization. It can be widely used in high-concurrency big data processing scenarios such as finance, e-commerce, and logistics.

[0020] 2. The improved batch processing engine's adaptive micro-batch adjustment mechanism dynamically adjusts the micro-batch duration based on the LSTM prediction model. Compared to the traditional fixed micro-batch architecture, it avoids resource waste caused by excessively small batches and prevents excessive latency caused by excessively large batches. The combination of data lineage tracing and distributed checkpoint technology reduces fault recovery time from tens of seconds to within 3 seconds, and achieves precise one-time semantic processing of data with no duplication or loss. This solves the pain points of data redundancy and poor consistency in the existing batch processing architecture, while supporting real-time processing and batch calibration collaboration, balancing business response speed and data accuracy.

[0021] 3. The two-layer grouping optimization algorithm, combined with a dynamic hierarchical storage strategy based on data popularity, breaks through the limitations of traditional static storage allocation. Through global node filtering and local optimal allocation, it achieves precise matching between data and storage nodes. The multi-replica redundant storage of hot data is linked with Redis caching, and the differentiated storage and compression of ordinary and cold data reduce storage costs by 41.7% and improve the query efficiency of hot data by 55.6%. The ultra-large data sharding parallel storage mechanism further improves the parallelism of reading and writing large-scale data, solving the problems of resource waste, hotspot congestion, and inefficient access in existing storage systems.

[0022] 4. The Prometheus+Grafana multi-dimensional monitoring system, in conjunction with the containerized elastic scaling mechanism, can capture system bottlenecks in real time and automatically trigger resource scheduling. Compared with the traditional manual intervention mode, the response speed is improved by more than 10 times, effectively avoiding node overload and excessive latency. The background low-priority defragmentation and index optimization design continuously improves the performance of the storage system without affecting real-time business, enabling the system to run stably and efficiently for a long time, reducing operation and maintenance costs and the complexity of manual intervention.

[0023] In summary, this invention, through technological innovation across the entire process of access, processing, storage, and monitoring, forms a complete solution for high-concurrency big data real-time processing and distributed storage optimization. It can be widely applied in fields such as finance, e-commerce, and logistics that have high requirements for concurrency, real-time performance, and data security, and has strong practicality and promotional value.

[0024] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0025] Figure 1 This is a flowchart of this embodiment. Detailed Implementation

[0026] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0027] It should be noted that when a component is described as "fixed to" another component, it can be directly on the other component or may have a component in between. When a component is considered "connected to" another component, it can be directly connected to the other component or may have a component in between. When a component is considered "set on" another component, it can be directly set on the other component or may have a component in between. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0029] This embodiment is applied to a real-time data analysis scenario for financial transactions. It needs to process 100,000 to 500,000 heterogeneous transaction data streams per second (including bank card transactions, third-party payments, wealth management redemptions, etc.), and requires core transaction data processing latency ≤ 500ms, data accuracy of 100%, support daily TB-level data storage and high-concurrency queries, and cope with sudden traffic surges (such as traffic peaks in scenarios like e-commerce promotions and wealth management product launches).

[0030] I. System Hardware Deployment Architecture This embodiment adopts a distributed cluster deployment, which is divided into an access layer cluster, a processing layer cluster, a storage layer cluster, and monitoring layer nodes. The specific configuration is as follows: 1. Access Layer Cluster: Deploy 8 edge access servers, configured with CPU Intel Xeon Gold 6348, memory 128GB, network card 100Gbps, distributed across three major nodes in Beijing, Shanghai, and Guangzhou, with 2-3 servers deployed on each node to enable data streams from multiple regions to access the nearest available location; 2. Processing Layer Cluster: Deploy 12 processing node servers, configured with Intel Xeon Platinum 8375C CPU (24 cores and 48 threads), 256GB of memory, and 2TB of SSD hard disk. Among them, 2 are master nodes (responsible for task scheduling) and 10 are slave nodes (responsible for data processing). Resources are managed based on YARN cluster. 3. Storage Layer Cluster: Adopts a hybrid storage architecture, including 3 types of nodes: ① 6 hot storage nodes (4 SSDs x 4, 128GB memory), ② 10 ordinary storage nodes (2 SSDs x 2 + 2 SATA x 8TB, 64GB memory), ③ 4 cold storage nodes (4 SATA x 12TB, 32GB memory). It is optimized based on the HDFS distributed file system and integrates a Redis cluster (3 masters and 6 slaves) as a hot data cache. 4. Monitoring layer nodes: Deploy 2 monitoring servers (CPU Xeon Gold 6326, memory 64GB, hard disk SSD 1TB) in master-slave mode to ensure uninterrupted monitoring service.

[0031] II. Software Selection and Core Module Implementation This embodiment is based on the Linux CentOS 8.5 operating system. The core software and components selected are as follows: the message middleware uses Kafka 3.4.0 (deployed with 3 broker nodes, 3 replicas) and RabbitMQ 3.12.0 (used for low-latency core business); the processing engine is based on Spark Streaming 3.4.1 and optimized; the storage system is based on HDFS 3.3.6 and optimized, and the cache uses Redis 7.0.12; the monitoring component uses Prometheus 2.45.0 + Grafana 10.2.0; the containerization technology uses Docker 24.0.7 + Kubernetes 1.28.3 to achieve elastic scaling of resources.

[0032] The specific implementations of each core module are as follows: (I) Implementation of Data Access Optimization Module The multi-level data access gateway was developed using Java and employs the Netty framework to implement high-concurrency network communication. The specific functionalities of each layer are implemented as follows: 1. Edge Access Layer: Adapts to both Kafka and RabbitMQ protocols. Core business data streams such as bank card transactions and wealth management redemption are accessed through RabbitMQ (the exchange type is Direct, and the routing key is divided according to the business type), while non-core streams such as third-party payments and log data are accessed through Kafka (the topic is divided into topic_core and topic_noncore, with 16 and 8 partitions respectively). Access nodes are allocated through a consistent hashing algorithm to avoid single-node overload. 2. Traffic Control Layer: The token bucket algorithm initially sets the token issuance rate to 100,000 / second. Traffic peak and fluctuation coefficient (fluctuation coefficient = current traffic / average traffic) are calculated every 100ms. When the fluctuation coefficient is ≥2, the token issuance rate increases by 10% per instance, up to a maximum of 500,000 / second. Data priority weights are divided into three levels: core business (weight 3), ordinary business (weight 2), and non-business data (weight 1). Tokens are preferentially allocated to high-weight data. When traffic exceeds the preset threshold (normal threshold 100,000 / second) by 50 times, an elastic buffer queue (implemented based on LinkedBlockingQueue, with capacity dynamically expanded to 1 million records) is triggered. When the queue is full, a degradation strategy is triggered, discarding only non-business data. 3. Data Preprocessing Layer: A parallel parsing algorithm (32 core threads in the thread pool, 64 maximum threads) is used to standardize and convert JSON, XML, and CSV data streams (to Parquet format). Redundant data filtering uses field hashing for deduplication (retaining the first occurrence of data, with a deduplication window of 5 seconds). At the same time, data integrity is verified (data with a core field missing rate >5% is discarded directly, and data with a missing rate <5% is filled with default values ​​and marked), achieving the cleaning and format conversion of 1.5 million data points per second.

[0033] (II) Implementation of Integrated Batch Processing Module The improved micro-batch processing engine is based on Spark Streaming optimizations. The core improvements are the adaptive batch size adjustment mechanism and enhanced data lineage tracking. Specific implementation details are as follows: 1. Adaptive Micro-batch Partitioning: An LSTM prediction model (input features are throughput, latency, and data volume of the past 10 batches; output is the optimal duration for the next batch) predicts changes in the data flow. The initial micro-batch duration is set to 1 second. When the predicted throughput increases by more than 30%, the duration is shortened to 0.5 seconds; when the latency exceeds 400ms, the duration is extended to 2 seconds. The adjustment range is strictly controlled within 0.1-10 seconds. Simultaneously, a batch data volume threshold is set (maximum data volume of 512MB per batch) to prevent excessively large batches from causing latency exceedances. 2. Data lineage tracking: Based on Apache Atlas, lineage information is collected, recording the source node, processing node, processing rules, storage location, and other information for each piece of data, generating a unique lineage ID (composed of timestamp + data hash value). Combining the RDD lineage mechanism and distributed checkpoint technology (checkpoints are saved every 30 seconds and stored in the / checkpoint path of HDFS), in the event of a failure, the range of lost data can be located by the lineage ID, and only the incremental data during the failure period is reprocessed, keeping the failure recovery time within 3 seconds, and the core business data processing latency is stabilized at 350-480ms; 3. Stream-Batch Collaboration: Real-time stream processing is responsible for core transaction data analysis with sub-second response time, while batch processing (based on SparkSQL) is responsible for full verification and statistical summary of the previous day's data every morning. Both share data storage directory and metadata information to avoid data redundancy and achieve a collaborative mode of "real-time processing + batch calibration" to ensure accurate semantic processing of data.

[0034] (III) Implementation of Distributed Storage Scheduling Module The storage system is built based on optimized HDFS and Redis caching. The two-layer grouping optimization algorithm is implemented in Python. The specific process and parameters are as follows: 1. Data Feature Extraction: The processed data is classified according to features. The key-value distribution features are calculated using MD5 hashing. The popularity attribute is divided according to the access frequency (hot data: access frequency ≥ 10 times / minute, ordinary data: 1-10 times / minute, cold data: < 1 time / minute). At the same time, information such as data size, creation time, and business line to which it belongs is recorded. The feature extraction cycle is 1 minute. 2. Execution of the two-level grouping optimization algorithm: Global node screening phase: Set screening thresholds (remaining storage capacity ≥ 20%, CPU utilization ≤ 70%, network bandwidth ≥ 10Gbps) to screen candidate nodes from 16 storage nodes, with the number of candidate nodes controlled between 8 and 10. Local optimal allocation stage: Construct the objective function F=α×transmission cost+β×storage cost+γ×access latency (α=0.4, β=0.3, γ=0.3, weights calibrated through multiple experiments), calculate the function value of each candidate node, and select the node corresponding to the minimum value as the optimal storage node; 3. Data storage strategy: Hot data (such as transaction data within the last hour) is stored in Redis cache + hot storage nodes (3 replicas, stored on nodes in different racks to avoid rack failures), ordinary data is stored in ordinary storage nodes (2 replicas), and cold data (historical data exceeding 30 days) is automatically migrated to cold storage nodes (1 replica, using compressed storage with a compression ratio of 4:1); very large data (single file > 10GB) is divided into 16 shards by key-value hash, with shard sizes evenly distributed between 512MB and 1GB to achieve parallel storage and retrieval, and shard information is recorded in the metadata server.

[0035] (iv) Implementation of the monitoring and optimization module The monitoring system is built using a Prometheus + Grafana architecture, and the dynamic optimization logic is implemented through a combination of shell scripts and Java interfaces. 1. Metric Collection and Monitoring: Prometheus collects core metrics every 10 seconds, with key monitoring metrics and threshold settings as follows: UnderReplicatedPartitions≤2, RecordsLagMax≤1000 records, storage IO idle rate≥30%, node CPU utilization≤75%, and data processing latency≤500ms; Grafana constructs a visualization dashboard, displaying metrics categorized by access layer, processing layer, and storage layer, while also setting alarm rules (triggering email and SMS alarms within 5 seconds after a metric exceeds the limit). 2. Dynamic resource scheduling: Based on Kubernetes, containerized elastic scaling is implemented. Scaling rules are set (if CPU utilization is ≥75% for 30 consecutive seconds, add 2 processing node containers) and scaling rules are set (if CPU utilization is ≤30% for 5 consecutive minutes, reduce 1 processing node container). Task migration is achieved through YARN's dynamic resource scheduling interface, which migrates tasks from overloaded nodes to nodes with lower load, ensuring that data processing is not interrupted during the migration process. 3. Storage optimization: Defragmentation is performed in the background using a low-priority thread (thread priority set to 3) from 2-4 AM daily (off-peak hours) to clean up storage fragments (merging files with fragment size <64MB into larger files). At the same time, HDFS indexes are optimized (rebuilding indexes based on Bloom filters to improve query hit rate). After optimization, data reading efficiency is improved by more than 40%.

[0036] III. Experimental Verification and Effect Analysis To verify the high concurrency processing capability, real-time performance, and storage optimization effect of this system, a comparative experimental environment was set up. The control group used the traditional Spark Streaming + standard HDFS architecture, while the experimental group used the system of this invention. The test scenarios and results are as follows: 1. High-concurrency traffic test: Simulate 500,000 concurrent data streams per second for 30 minutes. The control group experienced data loss (loss rate 2.3%) and excessive processing latency (average latency 820ms). The experimental group had no data loss, with an average processing latency of 420ms and a peak latency of 480ms. The token bucket + priority scheduling mechanism effectively ensured the processing of core data. 2. Storage Efficiency Test: Storing 1TB of financial transaction data, the control group occupied 1.2TB of storage space, and the average query time for hot data was 1.8 seconds; the experimental group, after tiered storage and compression optimization, occupied 0.7TB of storage space, and the average query time for hot data was 0.8 seconds, resulting in a 41.7% reduction in storage cost and a 55.6% improvement in query efficiency. 3. Fault recovery test: Simulated processing node failure (shutting down 1 processing node). The control group had a fault recovery time of 12 seconds and a data duplication rate of 1.5%; the experimental group had a fault recovery time of 2.8 seconds and no data duplication. The data lineage tracking and checkpoint mechanism effectively improved the fault recovery capability.

[0037] Experimental results show that the system can achieve real-time and accurate data processing in high-concurrency scenarios. At the same time, storage optimization reduces storage costs and improves data access efficiency, which has significant advantages over traditional architectures.

[0038] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A method for supporting high-concurrency real-time big data processing and distributed storage optimization, characterized in that: Includes the following steps: S1: Construct a multi-level data access gateway to preprocess multi-source heterogeneous concurrent data streams. The preprocessing includes dynamic rate limiting based on traffic characteristics, data format standardization, and redundant data filtering. The dynamic rate limiting adopts a combination of token bucket algorithm and data priority weight allocation to prioritize the passage of core business data streams. When the concurrent traffic exceeds the preset threshold by 10-100 times, an elastic buffer queue mechanism is triggered to avoid data loss. S2: Adopts a stream-batch integrated processing architecture. Through an improved micro-batch processing engine, the continuous data stream is divided into micro-batch units with dynamically adjustable durations. The duration of the micro-batch is adaptively adjusted within the range of 0.1-10 seconds according to the real-time throughput and latency requirements. At the same time, a data lineage tracking mechanism is introduced to achieve accurate one-time semantic processing of data after a fault, ensuring that the data is not duplicated or lost. S3: Distributed storage scheduling is performed based on data characteristics and node load status. The key-value distribution characteristics, hot attributes and access frequency of the processed data are extracted. The optimal storage node is allocated to the data through a two-layer grouping optimization algorithm. The dynamic mapping relationship between data and storage nodes is maintained. Hot data is stored with multiple replicas. The number of replicas is dynamically adjusted to 2-5 according to the access frequency. Cold data is automatically migrated to low-cost storage nodes. S4: Establish a closed loop of real-time monitoring and dynamic optimization, collect data processing latency, node load rate, storage IO rate and other indicators in real time. When node load imbalance or processing latency exceeds the preset threshold is detected, task migration and resource expansion are automatically triggered. At the same time, periodic defragmentation and index optimization of storage data are performed to improve data reading efficiency.

2. The method according to claim 1, characterized in that, The multi-level data access gateway mentioned in step S1 includes an edge access layer, a traffic control layer, and a data preprocessing layer. The edge access layer adopts a distributed deployment mode and supports multi-message middleware protocol adaptation such as Kafka and RabbitMQ. The traffic control layer dynamically adjusts the token bucket issuance rate by real-time statistics of the peak traffic and fluctuation coefficient of each data source. The data preprocessing layer adopts a parallel parsing algorithm to achieve format conversion and cleaning of millions of data points per second.

3. The method according to claim 1, characterized in that, The improved micro-batch processing engine described in step S2 is based on Spark Streaming optimization and introduces an adaptive batch size adjustment mechanism. It predicts changes in subsequent data stream throughput through a predictive model and adjusts the micro-batch unit size in advance. At the same time, it combines the RDD lineage mechanism and distributed checkpoint technology to control the fault recovery time within seconds and the core business data processing latency does not exceed 500 milliseconds.

4. The method according to claim 1, characterized in that, The two-layer grouping optimization algorithm described in step S3 includes two stages: global node screening and local optimal allocation. In the global node screening stage, candidate storage nodes are selected based on the remaining storage capacity of the nodes, CPU utilization, and network bandwidth. In the local optimal allocation stage, the optimal storage scheme is generated with data transmission cost, storage cost, and access latency as objective functions. At the same time, a data sharding strategy is adopted to divide the ultra-large data into multiple shards by key-value hash to achieve parallel storage and retrieval.

5. The method according to claim 1, characterized in that, The real-time monitoring system described in step S4 adopts a Prometheus+Grafana architecture, focusing on monitoring the number of asynchronous partitions in UnderReplicatedPartitions, the maximum consumption latency in RecordsLagMax, and the storage IO idle rate. Resource expansion adopts a containerized elastic scaling mode, supporting a combination of automatic expansion based on preset rules and manual intervention. Defragmentation is performed in the background using a low-priority thread to avoid affecting real-time data processing performance.

6. A device supporting high-concurrency real-time big data processing and distributed storage optimization, characterized in that: include: The data access optimization module is used to build a multi-level data access gateway to achieve dynamic rate limiting, format standardization and redundant data filtering for multi-source heterogeneous concurrent data streams, adapt to multiple message middleware protocols, and ensure stable data access in high-concurrency scenarios. The integrated stream and batch processing module is used to achieve adaptive micro-batch partitioning and processing of data streams through an improved micro-batch processing engine. Combined with data lineage tracking and fault recovery mechanisms, it ensures the accuracy and real-time performance of data processing and supports unified processing and integration of stream and batch data. The distributed storage scheduling module is used to allocate storage nodes based on data characteristics and node load status through a two-layer grouping optimization algorithm, maintain the dynamic mapping relationship between data and nodes, realize multi-replica storage of hot data and tiered storage of cold data, and optimize storage resource utilization. The monitoring and optimization module is used to collect system operation indicators in real time, trigger dynamic resource scheduling and storage optimization operations, perform data fragmentation defragmentation, index optimization and fault warning, and ensure the continuous and efficient operation of the system.

7. The apparatus according to claim 6, characterized in that, The data access optimization module has a built-in traffic prediction unit that analyzes historical traffic data through machine learning algorithms, predicts the arrival time of traffic peaks, and adjusts the rate limiting strategy and buffer queue capacity in advance. The integrated stream and batch processing module supports seamless integration with SparkSQL and MLlib components to achieve collaborative execution of real-time data analysis and machine learning tasks.

8. An electronic device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps of the method for supporting high-concurrency real-time big data processing and distributed storage optimization as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for supporting high-concurrency real-time big data processing and distributed storage optimization as described in any one of claims 1 to 5.

Images