Kafka storage and calculation separated data processing method and device

Abstract

The embodiment of the invention provides a kafka storage and calculation separated data processing method and device, and the method comprises the steps: constructing a multi-layer data processing architecture which comprises four layers: network communication, task distribution, processing scheduling and storage. Cold and hot data separation is realized through hierarchical design of a log pre-writing area and a main storage area, and intelligent scheduling of system resources is realized in combination with a copy manager, a coordinator and a load predictor. An adaptive compression algorithm and a verification mechanism are adopted to ensure the data transmission efficiency and integrity, a message cache region is created to store hotspot data, and flexible access of cold and hot data is realized based on message sites. According to the method, the defects of the traditional technology in storage and calculation coupling, storage efficiency, load balancing and the like are effectively overcome, and the performance and expandability of a message queue system are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and specifically to a data processing method and device for separating storage and computing in Kafka. Background Art

[0002] Existing Kafka data processing methods have obvious deficiencies. In the traditional architecture, the coupling degree between storage and computing is high, it is difficult to achieve elastic expansion of resources, and there is also the problem of uneven data access hotspots.

[0003] In addition, there are bottlenecks in the existing technology in terms of data storage efficiency. Most systems fail to effectively distinguish between hot and cold data, lack intelligent data compression and caching mechanisms, resulting in low storage space utilization and large access latency.

[0004] Existing systems have technical shortcomings in terms of load balancing and fault tolerance mechanisms. Lack of dynamic partition adjustment and prediction capabilities, it is difficult to cope with sudden traffic and system failures. Solving these problems is of great significance for improving the performance and reliability of the message queue system. Summary of the Invention

[0005] Aiming at the problems in the existing technology, this application provides a data processing method and device for separating storage and computing in Kafka, which can effectively solve the deficiencies of traditional technologies in terms of storage-computing coupling, storage efficiency, and load balancing, and significantly improve the performance and scalability of the message queue system.

[0006] To solve at least one of the above problems, this application provides the following technical solutions: In a first aspect, this application provides a data processing method for separating storage and computing in Kafka, including: Construct a multi-layer data processing architecture, the multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing and scheduling layer, and a storage layer. The storage layer includes a stream storage repository, the stream storage repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing and scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate; Receive batch message data sent by a producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a check code, write the compressed message data into the log shards in the write-ahead log area. The log shards are stored using solid-state drives. Establish a master-slave replication relationship between multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shards to the main storage area; Create a message buffer which is used to store hot message data, adjust the number of message partitions according to the prediction result of the load predictor, and a consumer reads the hot message data from the message buffer based on the message position. When the target message data does not exist in the message buffer, the consumer obtains cold data from the main storage area according to the message position and decompresses it. The coordinator records the consumption progress of the consumer and verifies the data integrity based on the checksum.

[0007] Further, it also includes: dividing the data processing architecture into call chains, setting up socket listeners in the network communication layer to process client connection requests, creating a message transmission channel and configuring transmission parameters, building a routing table in the task distribution layer to map the corresponding relationship between processors and message types, establishing a task queue, setting up a task priority scheduling mechanism, and distributing tasks to the corresponding processors according to the message types; Deploy a replica manager in the processing and scheduling layer to implement partition master-slave replication, configure a coordinator to manage partition allocation and consumer groups, deploy a load predictor to collect system resource usage data, create a stream repository in the storage layer and divide it into a write-ahead log area and a main storage area. The write-ahead log area uses a solid-state drive storage medium, and the main storage area uses an object storage service as the persistent storage.

[0008] Further, it also includes: calculating the number of log shards based on the total capacity of the write-ahead log area, assigning a unique identifier to each log shard, organizing the log shards into a linked list structure in chronological order, setting message offsets and timestamp indexes in the log shards, configuring the reserved space threshold of the log shards, and establishing a mapping relationship between the log shards and physical storage devices. The main storage area divides storage buckets according to time ranges and creates metadata indexes for each storage bucket; Configure the master-slave replication parameters of the replica manager, establish a synchronization mechanism for partition replicas, build a partition allocation table and a consumer group management table in the coordinator, record consumer group member information and consumption progress. The load predictor collects system resource metrics such as processor utilization rate, memory occupancy rate, and disk read / write rate, constructs a time series prediction model to calculate the resource usage trend, and generates a load balancing strategy based on the prediction result.

[0009] Further, it also includes: parsing the message headers and message bodies of the batch message data sent by the producer, statistically analyzing the data distribution characteristics of the message bodies, calculating data characteristic metrics such as the entropy value, repetition degree, and numerical range of the message bodies, establishing a compression algorithm evaluation matrix, calculating the compression ratio and processing delay of different compression algorithms according to the data characteristic metrics, and selecting the optimal compression algorithm in the evaluation matrix; Apply the selected compression algorithm to the batch message data to generate a compressed byte stream, calculate the CRC32 checksum of the compressed byte stream, package the compressed data length, compression algorithm identifier, and checksum into a metadata header, assemble the metadata header and the compressed byte stream into a data block, and select an available log shard in the write-ahead log area to write the data block.

[0010] Further, it also includes: allocating solid-state drive storage space for the log shard, configuring the storage parameters of the log shard, setting the storage buffer size and the disk flushing policy, selecting slave nodes based on the configuration information of the replica manager, establishing a replication channel between the master and slave nodes, sending the log shard data to the slave nodes in batches, and after verifying the data integrity at the slave nodes, returning an acknowledgment message to the master node; Establish a message data upload task queue, scan the confirmed replicated data area in the log shard at a preset time interval, group and package the messages in the data area according to the time range, calculate the storage path of the data packet, asynchronously upload the data packet to the storage bucket corresponding to the main storage area through the object storage interface, and update the storage location index of the data packet.

[0011] Further, it also includes: allocating memory space for the message buffer, constructing a cache index table to record the mapping relationship between the message position and the storage address, setting the capacity limit and the eviction policy of the buffer, establishing a hot message scoring mechanism, calculating the message heat score based on the access frequency of the message and the time decay factor, and loading the message data with a heat score exceeding the preset threshold into the buffer; Monitor the read and write load data of the message partition. The load predictor trains a prediction model based on historical load data, calculates the load trend within a future time window, triggers partition expansion when the predicted load exceeds the preset threshold, redistributes the message data in the existing partition to the newly added partition according to the consistent hashing algorithm, and updates the partition routing table.

[0012] Further, it also includes: the consumer sends a position query request to the coordinator, the coordinator returns the storage location information corresponding to the message position, the consumer searches for the target message data in the message buffer according to the storage location information. If the target message data does not exist in the buffer, it obtains the corresponding data packet from the main storage area, reads the metadata header in the data packet to parse the compression algorithm identifier, and calls the corresponding decompression method to restore the message data; Calculate the checksum of the decompressed message data and compare it with the checksum recorded in the metadata header. After passing the verification, return the message data to the consumer. After the consumer processes the message, submit the consumption position to the coordinator, and the coordinator records the consumption position in the position management table and periodically persists the position management table to the storage system.

[0013] In a second aspect, the present application provides a kafka storage and computing separation data processing device, including: An architecture building module for building a multi-layer data processing architecture. The multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing and scheduling layer, and a storage layer. The storage layer includes a stream repository, and the stream repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing and scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate; A message storage module for receiving batch message data sent by a producer, calculating data characteristics of the batch message data, selecting a compression algorithm according to the data characteristics, performing compression encoding on the batch message data to generate a check code, writing the compressed message data into a log shard in the write-ahead log area. The log shard is stored using a solid-state drive, establishing a master-slave replication relationship among multiple log shards according to the configuration of the replica manager, and asynchronously uploading the message data in the log shard to the main storage area; A message caching module for creating a message cache area. The message cache area is used to store hot message data, adjusting the number of message partitions according to the prediction result of the load predictor. A consumer reads the hot message data from the message cache area based on a message position. When the target message data does not exist in the message cache area, the consumer obtains cold data from the main storage area according to the message position and decompresses it. The coordinator records the consumption progress of the consumer and verifies data integrity based on the check code.

[0014] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the kafka storage-computation separation data processing method are implemented.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the kafka storage-computation separation data processing method are implemented.

[0016] In a fifth aspect, the present application provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the kafka storage-computation separation data processing method are implemented.

[0017] As can be seen from the above technical solution, the present application provides a data processing method and device for Kafka storage-computation separation. By constructing a multi-layer data processing architecture, including four layers: network communication, task distribution, processing scheduling, and storage. The separation of hot and cold data is achieved through the hierarchical design of the write-ahead log area and the main storage area, and the intelligent scheduling of system resources is realized by combining the replica manager, coordinator, and load predictor. The adaptive compression algorithm and verification mechanism are adopted to ensure the data transmission efficiency and integrity. A message buffer is created to store hot data, and flexible access to hot and cold data is realized based on the message position. This method effectively solves the deficiencies of traditional technologies in aspects such as storage-computation coupling, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 It is a schematic flowchart of the data processing method for Kafka storage-computation separation in the embodiments of the present application; Figure 2 It is a structural diagram of the data processing device for Kafka storage-computation separation in the embodiments of the present application; Figure 3 It is a schematic structural diagram of the electronic device in the embodiments of the present application.

[0020] Reference Numerals: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present application belong to the scope of protection of the present application.

[0022] In the technical solution of this application, the acquisition, storage, use, processing, etc. of data all comply with the relevant provisions of national laws and regulations.

[0023] Considering the problems existing in the prior art, this application provides a data processing method and device for separating storage and computing in Kafka. By constructing a multi-layer data processing architecture, including four layers: network communication, task distribution, processing scheduling, and storage. Through the hierarchical design of the write-ahead log area and the main storage area, hot and cold data separation is achieved. Combining the replica manager, coordinator, and load predictor to realize the intelligent scheduling of system resources. Using an adaptive compression algorithm and a verification mechanism to ensure data transmission efficiency and integrity, creating a message buffer to store hot data, and realizing flexible access to hot and cold data based on message positions. This method effectively solves the deficiencies of traditional technologies in aspects such as storage-computation coupling, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system.

[0024] In order to effectively solve the deficiencies of traditional technologies in aspects such as storage-computation coupling, storage efficiency, and load balancing, and significantly improve the performance and scalability of the message queue system, this application provides an embodiment of a data processing method for separating storage and computing in Kafka. See Figure 1 , the data processing method for separating storage and computing in Kafka specifically includes the following content: Step S101: Construct a multi-layer data processing architecture. The multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing scheduling layer, and a storage layer. The storage layer includes a stream storage repository. The stream storage repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate; Optionally, based on the architecture characteristics of Kafka, this embodiment constructs a data processing architecture with clear hierarchical divisions. In the network communication layer, a high-performance network communication is implemented using an asynchronous IO framework based on Netty. Multiple processors are organized through a pipeline mechanism, including a codec, an idle detector, and a compression processor. Each processor independently completes a specific function, and the processors cooperate with each other through an event transmission mechanism to achieve high-concurrency processing capabilities at the network level.

[0025] This embodiment implements an intelligent routing mechanism at the task distribution layer. By using the consistent hashing algorithm to construct a message partition routing table, it ensures the uniform distribution of messages among partitions. The routing table adopts a two-layer structure. The first layer is the mapping from topics to partitions, and the second layer is the mapping from partitions to processors. When partition expansion or contraction is required, through the rebalancing operation of the consistent hash ring, the affected messages are re-routed to minimize the data migration cost.

[0026] This embodiment designs an innovative processing scheduling layer structure. The replica manager adopts a consistent replication mechanism based on the Raft protocol and ensures data consistency through the Write-Ahead Log (WAL). Each partition maintains an independent replication group, and the members within the group detect the node status through the heartbeat mechanism. When the primary node fails, a new primary node is elected through the election algorithm. The coordinator is responsible for the management of the consumer group and implements the consumer balancing algorithm to ensure the balanced distribution of partition loads to consumers.

[0027] This embodiment implements a deep learning model based on LSTM (Long Short-Term Memory) in the load predictor. This model receives multi-dimensional time series data as input, including metrics such as CPU usage rate, memory occupancy rate, disk I / O rate, and network bandwidth. Through a multi-layer LSTM network, it captures the time-dependent relationships of these metrics and predicts the resource usage trends within a future time window. The prediction results are used to guide the decision-making of partition expansion and contraction, effectively preventing system overload.

[0028] This embodiment creates a two-layer storage structure in the storage layer. The Write-Ahead Log area uses solid-state drives for storage and achieves high-speed writing through the zero-copy technology. The size of the log shards is dynamically adjusted according to the expected throughput, and each shard contains an index file and a data file. The index file records the mapping from message offsets to physical locations, supporting fast message positioning. The data file uses the append-write method to ensure sequential I / O performance. The main storage area realizes data persistence through the object storage service and adopts a hierarchical storage strategy, with hot data preferentially stored in the storage layer with higher performance.

[0029] This embodiment implements an efficient data synchronization mechanism. The main storage area adopts an asynchronous upload strategy, sets up a data upload task queue, and batches the data in the Write-Ahead Log area to the object storage at predefined time intervals. During the upload process, a multi-level caching strategy is used to cache frequently accessed data blocks in memory to reduce the access frequency of the object storage. At the same time, an intelligent data preheating mechanism is implemented to predict the data blocks that may be accessed according to the access pattern and load them into the cache in advance.

[0030] This embodiment constructs a complete monitoring and alerting system. Standardized metric collection points are established between layers to monitor the running status of the system in real time. When abnormal situations are detected, such as network latency exceeding the threshold, insufficient storage space, etc., the operation and maintenance personnel are notified through a multi-level alerting mechanism. At the same time, an automated fault recovery mechanism is implemented, and recovery operations can be automatically executed for common fault scenarios.

[0031] Through the above technological innovations, this embodiment effectively solves the key problems existing in the traditional Kafka architecture: poor scalability, insufficient data reliability, low resource utilization, etc. caused by the coupling of storage and computing. In practical applications, this solution can support large-scale distributed message processing scenarios, and significantly improves the scalability and resource utilization of the system through a storage-computation separation architecture. It is particularly suitable for scenarios with high requirements for data reliability and processing performance, such as finance and e-commerce. Through a multi-layer architecture design and intelligent scheduling strategies, the overall performance of the message processing system is improved.

[0032] Step S102: Receive the batch message data sent by the producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a check code, write the compressed message data into the log shard in the write-ahead log area, the log shard is stored using a solid-state drive, establish a master-slave replication relationship between multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shard to the main storage area; Optionally, this embodiment adopts a batch processing mechanism for message reception and processing. The producer organizes messages using a ProducerBatch object and controls the data volume of a single batch of messages through a configurable batch size parameter. The message reception buffer adopts a circular buffer design, and the writing and extraction of data are controlled by read and write pointers. When the data volume in the buffer reaches the batch threshold or the waiting time exceeds the configured delay threshold, the processing flow of batch messages is triggered.

[0033] This embodiment implements an adaptive data characteristic analysis mechanism. Multi-dimensional feature extraction is performed on batch message data, including repeated pattern analysis, entropy value calculation, and numerical distribution statistics. Repeated patterns are identified through the LZ77 algorithm with a sliding window, and the occurrence frequency of repeated substrings in the data is calculated.

[0034] The entropy value H is calculated by the formula H = -∑(Pi×log2(Pi)), where Pi represents the probability of different byte values occurring. The lower the entropy value, the stronger the compressibility of the data. The numerical distribution is statistically analyzed using a histogram to identify the concentration and dispersion of the data.

[0035] This embodiment designs an intelligent compression algorithm selection strategy. A compression algorithm feature matrix is established, which includes common compression algorithms such as LZ4, Snappy, and ZSTD. For each algorithm, its compression ratio and compression speed metrics under different data characteristics are recorded. Through a multi-factor decision-making model, considering data characteristics, algorithm performance, and system resource status comprehensively, the optimal compression algorithm is selected. For example, for log data with significant repetitive patterns, the LZ4 algorithm is preferentially selected; for time-series data with concentrated numerical distributions, a compression algorithm specifically for numerical values is preferentially selected.

[0036] This embodiment implements an efficient compression processing flow. The compression process uses multi-threaded parallel processing. After slicing the batch message data, the compression is executed in parallel. The compression result includes a compressed data block and a metadata header, and the metadata header records information such as the compression algorithm type, the original data size, and the compressed size. At the same time, a CRC32 checksum is generated, and the checksum calculation covers the compressed data and the metadata header to ensure data integrity.

[0037] This embodiment constructs a reliable data writing mechanism. The write-ahead log area uses solid-state drive storage, and the asynchronous writing method is adopted to improve the writing performance. The writing of log slices adopts the append writing mode, and each slice maintains a write position pointer to ensure sequential writing. The memory-mapped file technology is used during the writing process to reduce system call overhead. At the same time, the write buffer management is implemented. When the buffer utilization rate exceeds the threshold, an asynchronous disk flushing operation is triggered.

[0038] This embodiment designs an innovative master-slave replication mechanism. The replica manager maintains data consistency between the master and slave nodes through a Raft-based consensus protocol. After receiving a write request, the master node first writes the data to the local log slice, and then sends it to all slave nodes in parallel. The pipeline replication technology is adopted to send the next batch while waiting for the confirmation of the previous batch, improving the replication efficiency. After receiving the data, the slave node verifies the data integrity, writes the data to the local log slice, and returns a confirmation message to the master node.

[0039] This embodiment implements an intelligent data upload strategy. The main storage area uses an object storage service to support long-term data persistence. The data upload is asynchronous, and an upload task queue is set up to manage upload requests. The upload frequency is dynamically adjusted according to the usage of the write-ahead log area, and the upload frequency is increased when the log slice utilization rate is relatively high. The upload process supports resume from breakpoint. When the upload is interrupted due to network problems, it can continue to upload from the breakpoint position.

[0040] Through the above technological innovations, this embodiment effectively solves the key problems in message processing: low data compression efficiency, storage performance bottlenecks, insufficient data reliability, etc. In practical applications, this solution can adapt to different types of message data. Through intelligent compression strategies and reliable storage mechanisms, it significantly improves data processing efficiency and reliability. It is particularly suitable for scenarios such as large-scale log collection and time-series data processing. Through a multi-level storage architecture and an asynchronous processing mechanism, the overall performance of the message processing system is improved. The systematicness and innovativeness of this solution enable it to adapt to the application requirements of enterprises of different scales. Through continuous optimization and improvement, a comprehensive improvement in message processing efficiency is achieved.

[0041] Step S103: Create a message buffer. The message buffer is used to store hot message data. Adjust the number of message partitions according to the prediction result of the load predictor. The consumer reads the hot message data from the message buffer based on the message position. When the target message data does not exist in the message buffer, the consumer obtains cold data from the main storage area according to the message position and decompresses it. The coordinator records the consumption progress of the consumer and verifies the data integrity based on the check code.

[0042] Optionally, this embodiment implements a multi-level cache management architecture based on message access characteristics. The message buffer adopts an off-heap memory design and directly allocates memory space through DirectByteBuffer, avoiding the limitations of JVM heap memory management. The buffer is divided into multiple memory pages of a fixed size, and each memory page maintains an independent memory mapping table to record the position information of the message in physical memory. Data is quickly read through the zero-copy technology, significantly reducing the memory copy overhead.

[0043] This embodiment designs an innovative hot message recognition mechanism. A sliding time window is used to count the access frequency of messages, and the window size can be dynamically adjusted according to business characteristics. Calculate the heat score for each message, and the score calculation formula is: Score = Frequency×Recency×Weight, where Frequency is the access frequency, Recency is the time decay factor, and Weight is the message priority weight. The natural decay process of message heat over time is simulated through an exponential decay function, accurately reflecting the real-time heat change of the message.

[0044] This embodiment implements an intelligent partition dynamic adjustment strategy. The load predictor analyzes historical load data through a deep learning model and establishes a time series prediction network based on LSTM. The input features include indicators such as partition read / write QPS, message backlog, and consumption latency. The multi-layer LSTM network learns the time series dependency relationships between these indicators. The model outputs the predicted load trend within a future time window. When the predicted load exceeds the preset threshold, a partition expansion operation is triggered.

[0045] This embodiment constructs an efficient message reading process. The consumer first obtains the consumption position information from the coordinator. The coordinator maintains a position mapping table for the consumer group, recording the consumption progress of each consumer. The consumer calculates the logical offset of the message based on the position information and quickly locates the position of the message in the buffer through multi-level lookups. If the target message is not in the buffer, the cold data loading process is started to obtain the corresponding compressed data packet from the main storage area.

[0046] This embodiment designs a reliable data integrity verification mechanism. During message decompression, first parse the metadata header of the data packet to obtain the compression algorithm type and checksum information. Select the corresponding decompression method according to the compression algorithm type. After decompression, calculate the CRC32 checksum of the message data and compare it with the checksum in the metadata header. If the verification fails, the retry mechanism is triggered to obtain the data from the storage system again to ensure data integrity.

[0047] This embodiment implements an optimized cache eviction strategy. An improved LRU-K algorithm is used to manage the cache space, recording the timestamps of the last K accesses of each message. The future access probability of the message is predicted more accurately through these access records. When the cache space is insufficient, the messages with lower access probabilities are preferentially evicted. At the same time, a preloading mechanism is implemented to predict the messages that may be accessed according to the access pattern of the messages and load them into the buffer in advance.

[0048] This embodiment constructs a complete consumption progress management mechanism. The coordinator periodically persists the consumption positions to the storage system, using a two-phase commit protocol to ensure the atomicity of position updates. The position information includes fields such as consumer group ID, topic partition, consumption position, and timestamp, and a version number mechanism is used to handle concurrent updates. When the consumer restarts or rebalances, the previous consumption progress can be quickly restored.

[0049] Through the above technological innovations, this embodiment effectively solves the key problems in distributed message systems, such as high access latency of hot messages, low cache utilization, and load imbalance. In practical applications, this solution can intelligently identify hot messages and optimize the access path. Through multi-level caching and dynamic partition adjustment, it significantly improves the throughput and response speed of message processing. It is particularly suitable for scenarios with significant hot access characteristics, such as social media hot events and e-commerce flash sales activities. Through precise hot spot identification and efficient cache management, it realizes the performance optimization of the message processing system. The systematicness and innovativeness of this solution enable it to adapt to different scales of message processing requirements, and through continuous optimization and improvement, it realizes the overall improvement of the message system.

[0050] As can be seen from the above description, the data processing method with separated storage and computing for Kafka provided by the embodiment of this application can build a multi-layer data processing architecture, including four layers: network communication, task distribution, processing scheduling, and storage. Through the hierarchical design of the write-ahead log area and the main storage area, hot and cold data separation is achieved. Combining the replica manager, coordinator, and load predictor realizes the intelligent scheduling of system resources. The adaptive compression algorithm and verification mechanism are used to ensure the data transmission efficiency and integrity. A message buffer is created to store hot data, and flexible access to hot and cold data is realized based on message positions. This method effectively solves the deficiencies of traditional technologies in aspects such as storage-computation coupling, storage efficiency, and load balance, and significantly improves the performance and scalability of the message queue system.

[0051] In an embodiment of the data processing method with separated storage and computing for Kafka of this application, it may also specifically include the following content: Step S201: Divide the data processing architecture into call chains. Set up socket listeners in the network communication layer to handle client connection requests, create a message transmission channel and configure transmission parameters. Build a routing table in the task distribution layer to map the corresponding relationship between processors and message types, establish a task queue, set up a task priority scheduling mechanism, and distribute tasks to the corresponding processors according to the message types. Step S202: Deploy a replica manager in the processing scheduling layer to implement partition master-slave replication, configure a coordinator to manage partition allocation and consumer groups, deploy a load predictor to collect system resource usage data. Create a stream repository in the storage layer and divide it into a write-ahead log area and a main storage area. The write-ahead log area uses a solid-state drive storage medium, and the main storage area uses an object storage service as persistent storage.

[0052] Optionally, this embodiment adopts a hierarchical call chain design to achieve full-process tracking of requests from access to processing. In the network communication layer, an event-driven framework is built using the Reactor pattern. The main Reactor thread is responsible for receiving client connection requests, and the sub-Reactor thread pool processes data reception and transmission for established connections. High-efficient IO multiplexing is achieved through the epoll mechanism, significantly improving the connection processing ability. Each connection is assigned a unique session ID to track the lifecycle of requests.

[0053] This embodiment implements an adaptive flow control mechanism in the message transmission channel. The transmission rate is dynamically adjusted through the sliding window algorithm, and the window size is automatically adjusted according to network latency and packet loss rate. Transmission parameters include TCP send buffer size, receive buffer size, heartbeat interval, etc., which can be dynamically optimized according to the actual network conditions. At the same time, message fragmentation transmission is implemented, slicing super-large messages to ensure transmission reliability.

[0054] This embodiment designs an efficient task routing mechanism. The routing table adopts a multi-level hash structure. The first level is hashed based on the message topic, the second level is hashed based on the message type, and finally mapped to specific processors. To improve the lookup efficiency, the routing table supports regular optimization, migrating frequently accessed routing entries to the fast access area. The processor pool adopts an elastic scaling design and can dynamically adjust the number of processors according to the load.

[0055] This embodiment constructs an intelligent task scheduling system. The task queue adopts a multi-level feedback queue design, and tasks with different priorities enter different queue levels. Priority calculation considers multiple factors: the timeliness requirement of the message, the user service level, the queue waiting time, etc. Through dynamic time slice allocation, it is ensured that high-priority tasks can be processed quickly, while preventing low-priority tasks from starving.

[0056] This embodiment implements a reliable replica management mechanism. The replica manager implements master-slave replication based on the improved Raft protocol, introducing parallel replication optimization, allowing slave nodes to write multiple log entries in parallel. The replication process adopts a pipelined design, sending the next batch of data while waiting for the confirmation of the previous batch. At the same time, an intelligent failover mechanism is implemented, which can quickly elect a new master node when the master node fails.

[0057] This embodiment designs an innovative partition management strategy. The coordinator maintains a global partition allocation table and uses the consistent hashing algorithm for partition allocation. The allocation process considers multiple constraints: broker load balancing, rack awareness, data locality, etc. When partition rebalancing occurs, the data transmission overhead is minimized through incremental migration. The consumer group management adopts a distributed session mechanism, ensuring quick detection and handling in case of consumer failures.

[0058] In this embodiment, an accurate load monitoring system is constructed. The load predictor collects system metrics through various channels such as JMX and cgroups, including CPU usage, memory occupancy, disk I / O, network throughput, etc. The collected data is stored in a time series database, and multi-dimensional metric aggregation analysis is established. The prediction model uses an LSTM neural network, and the input features include historical load data and system event logs, and the output predicts the resource usage trend within a future time window.

[0059] In this embodiment, an efficient storage management mechanism is implemented. The write-ahead log area utilizes the high IOPS characteristics of solid-state drives and adopts a multi-queue design to improve parallel write performance. The log files are sliced into fixed sizes, and memory mapping technology is used to accelerate read and write operations. The main storage area uses an object storage service to achieve heat tiered storage of data. Hot data is preferentially stored in the storage layer with higher performance, and cold data is automatically migrated to the storage layer with lower costs.

[0060] Through the above technological innovations, this embodiment effectively solves the key problems in the distributed messaging system: limited connection processing capacity, unbalanced task scheduling, insufficient data reliability, etc. In practical applications, this solution can support large-scale message processing requirements. Through a multi-layer architecture and intelligent scheduling, the throughput and reliability of the system are significantly improved. It is particularly suitable for scenarios such as financial transactions and Internet of Things data processing. Through flexible storage strategies and reliable replication mechanisms, the overall performance of the messaging system is improved. The systematicness and innovativeness of this solution enable it to adapt to the application requirements of enterprises of different scales. Through continuous optimization and improvement, a comprehensive upgrade of the message processing system is achieved.

[0061] In an embodiment of the data processing method for separating storage and computing of Kafka in this application, the following content may also be specifically included: Step S301: Calculate the number of log slices based on the total capacity of the write-ahead log area, assign a unique identifier to each log slice, organize the log slices into a linked list structure in chronological order, set message offsets and timestamp indexes in the log slices, configure the reserved space threshold of the log slices, and establish a mapping relationship between the log slices and physical storage devices. The main storage area divides storage buckets according to time ranges and creates metadata indexes for each storage bucket; Step S302: Configure the master-slave replication parameters of the replica manager, establish a synchronization mechanism for partition replicas, build a partition assignment table and a consumer group management table in the coordinator, record consumer group member information and consumption progress. The load predictor collects system resource metrics such as processor usage rate, memory occupancy rate, and disk read and write speed, constructs a time series prediction model to calculate the resource usage trend, and generates a load balancing strategy based on the prediction results.

[0062] Optionally, in this embodiment, a dynamic allocation strategy is adopted in log shard management. The initial number of shards is calculated by the ratio of the total capacity of the write-ahead log area to the reference size of a single shard. The shard size adopts a logarithmic growth pattern, and the size of the new shard is 1.5 times that of the previous shard. This design can adapt to the growth trend of message traffic. Each shard is assigned a 64-bit unique identifier, which contains timestamp, sequence number, and node ID information to ensure uniqueness in a distributed environment.

[0063] This embodiment implements an efficient shard index structure. Log shards are connected by a doubly linked list, and each shard node contains reference pointers to the previous and next shards. A sparse index is established inside the shard, and an index entry is created every 4KB of message data. The index entry contains message offset, physical location, and timestamp information. The skiplist structure is used to organize the index entries, achieving fast lookup with O(log n) time complexity. At the same time, the index cache of hot shards is maintained in memory to improve the lookup performance of frequent access.

[0064] This embodiment designs an innovative space management mechanism. The reserved space threshold is dynamically calculated, considering the historical write rate and the remaining capacity. When the shard utilization rate reaches the threshold, an asynchronous migration mechanism is triggered to migrate the old data to the main storage area. The migration process adopts a batch copy strategy, and the migration efficiency is improved by parallel transmission of multiple threads. At the same time, intelligent defragmentation is achieved, and small shards are merged regularly to optimize the storage space utilization.

[0065] This embodiment constructs reliable bucket management. The main storage area is divided into buckets according to the daily time range, and each bucket contains multiple data files. The metadata index of the bucket adopts a B+ tree structure, and the index entry contains information such as time range, message offset range, file path, etc. The index storage space is optimized by prefix compression technology, and at the same time, an incremental update mechanism of the index is implemented to reduce the index maintenance overhead.

[0066] This embodiment implements an accurate replica synchronization mechanism. The replica manager adopts an improved Raft protocol and introduces parallel replication optimization. The replication parameters include batch size, number of replication threads, heartbeat interval, etc., and these parameters can be dynamically adjusted according to the network conditions. The pipelining design is used in the synchronization process, allowing multiple batches of data to be sent simultaneously, significantly improving the replication efficiency. At the same time, an intelligent catch-up mechanism is implemented, and the lagging replicas can quickly synchronize the latest data.

[0067] This embodiment designs an innovative partition management strategy. The partition allocation table is implemented using a consistent hashing ring, and the virtual node technology is used to improve the balance of allocation. The consumer group management table records the session information, partition allocation status, and consumption positions of consumers. The position management adopts a two-phase commit protocol to ensure that it can be restored to a consistent state in case of failures. At the same time, an incremental adjustment of consumer rebalancing is implemented to minimize the impact of the rebalancing process on the service.

[0068] This embodiment constructs an intelligent load prediction system. Through multi-dimensional metric collection, a complete resource profile is established. The collected metrics include CPU usage rate, memory occupancy rate, disk I / O, etc. at the system level, as well as request queue length, processing latency, etc. at the application level. The prediction model uses an LSTM neural network, and the attention mechanism is used to capture the correlation between different metrics. The model training adopts a sliding window strategy, and the window size is dynamically adjusted according to the load change period.

[0069] This embodiment implements an adaptive load balancing strategy. Based on the prediction results, the load score is calculated, and the score calculation formula is: Score = w1 CPU + w2 Memory + w3 IO + w4 Network, where wi is the weight factor, and Network represents the network bandwidth rate. When the load score of a certain node exceeds the threshold, the load balancing operation is triggered. The balancing process considers the data locality principle and preferentially migrates the load to the node with data replicas to reduce the data transmission overhead.

[0070] Through the above technological innovations, this embodiment effectively solves the key problems in the distributed storage system: low storage space management efficiency, poor data synchronization performance, load imbalance, etc. In practical applications, this solution can support large-scale message storage and processing requirements. Through intelligent shard management and predictive load balancing, the performance and reliability of the system are significantly improved. It is particularly suitable for scenarios that require high throughput and low latency, such as real-time data analysis, log collection, etc. Through multi-level optimization strategies, the overall performance of the storage system is improved. The systematicness and innovativeness of this solution enable it to adapt to the application requirements of enterprises of different scales. Through continuous optimization and improvement, a comprehensive upgrade of the storage system is achieved.

[0071] In an embodiment of the kafka storage-computation separation data processing method of this application, it may also specifically include the following content: Step S401: Parse the message header and message body of the batch message data sent by the producer, count the data distribution characteristics of the message body, calculate data characteristic indicators such as the entropy value, repetition degree, and numerical range of the message body, establish a compression algorithm evaluation matrix, calculate the compression ratio and processing delay of different compression algorithms according to the data characteristic indicators, and select the optimal compression algorithm in the evaluation matrix; Step S402: Apply the selected compression algorithm to the batch message data to generate a compressed byte stream, calculate the CRC32 checksum of the compressed byte stream, package the compressed data length, compression algorithm identifier, and checksum into a metadata header, assemble the metadata header and the compressed byte stream into a data block, and select an available log shard in the write-ahead log area to write the data block.

[0072] Optionally, in this embodiment, the zero-copy technology is adopted in the message parsing phase to directly access the message data through memory mapping. The message header parsing adopts a fixed-length field design, including information such as message length, version number, topic, and partition, and quickly locates the positions of each field through displacement operations. The message body adopts variable-length coding and is stored in the TLV (Type-Length-Value) format, supporting nested structures and adapting to complex message formats. The parsing process realizes parallel processing, and multiple parsing threads process different message batches simultaneously.

[0073] This embodiment realizes accurate data characteristic analysis. The entropy value is calculated using the sliding window method, and the window size can be dynamically adjusted according to the message size.

[0074] The entropy value calculation formula is H = -∑(Pi×log2(Pi)), where Pi represents the probability of the byte value i appearing. The repetition degree analysis uses an improved LZ77 algorithm to identify repeated data segments by maintaining a sliding window of a fixed size. The numerical range analysis adopts histogram statistics, divides the numerical distribution into multiple intervals, and calculates the data density of each interval.

[0075] This embodiment designs an innovative compression algorithm evaluation mechanism. The evaluation matrix includes multiple dimensions: compression ratio, compression speed, decompression speed, memory occupancy, etc. Each dimension is assigned a weight coefficient, and the weight value can be dynamically adjusted according to the application scenario. For example, in a scenario with high real-time requirements, the weight of the compression speed will be increased accordingly. The evaluation process adopts the sample test method, and representative samples are extracted from the message batches for compression testing.

[0076] This embodiment constructs an adaptive algorithm selection strategy. According to the message characteristics and evaluation results, a decision tree model is established. The branch nodes of the decision tree include conditions such as entropy value threshold and repetition degree threshold, and the leaf nodes correspond to specific compression algorithms. By traversing the decision tree, the most suitable compression algorithm can be quickly selected. At the same time, an algorithm caching mechanism is realized, and the algorithm selection results of previous message batches with similar characteristics can be reused.

[0077] This embodiment realizes an efficient compression processing flow. After selecting the algorithm, a multi-threaded parallel compression strategy is adopted. The message batches are sliced according to a fixed size, and each thread is responsible for the compression task of one slice. The compression results are passed through a shared memory queue, avoiding data copying between threads. At the same time, dynamic adjustment of the compression level is realized, and an appropriate compression level is selected according to the CPU load situation.

[0078] This embodiment designs a reliable data verification mechanism. The CRC32 verification is implemented with hardware acceleration, using the CRC32 instructions of modern CPUs to significantly improve the calculation speed. The verification process covers the compressed data and the metadata header to ensure data integrity. At the same time, an incremental verification mechanism is realized. When a data block is partially modified, only the verification value of the modified part needs to be recalculated.

[0079] This embodiment constructs a complete metadata management. The metadata header adopts a compact binary format, and the storage overhead is reduced through bit field technology. The compression algorithm identifier is represented by an enumeration value, supporting fast algorithm identification. The data length field uses variable-length coding to adapt to data blocks of different sizes. At the same time, version control of the metadata is realized, supporting backward compatibility of the metadata format.

[0080] This embodiment realizes an optimized data writing strategy. The selection of log slices is based on multiple factors: the remaining space of the slice, the writing load, data locality, etc. The writing process is asynchronous. After accumulating multiple data blocks through a writing buffer, they are written in batches to improve the writing efficiency. At the same time, a write-ahead mechanism is realized. Before the data is actually written, the metadata of the data block is first written to the write-ahead log to ensure data persistence.

[0081] Through the above technological innovations, this embodiment effectively solves the key problems in message compression processing: inaccurate selection of compression algorithms, low compression efficiency, insufficient guarantee of data integrity, etc. In practical applications, this solution can intelligently select the optimal compression algorithm, and through parallel processing and optimized writing strategies, significantly improve the throughput and reliability of message processing. It is particularly suitable for large-scale message processing scenarios, such as log collection and monitoring data processing. Through multi-level optimization strategies, the overall performance of the message compression and storage system is improved. The systematicness and innovativeness of this solution enable it to adapt to different types of message data, and through continuous optimization and improvement, the overall message processing efficiency is improved.

[0082] In an embodiment of the data processing method for separating storage and computing in Kafka of this application, it may also specifically include the following content: Step S501: Allocate solid-state drive storage space for the log shard, configure the storage parameters of the log shard, set the storage buffer size and disk flushing policy, select slave nodes based on the configuration information of the replica manager, establish a replication channel between the master and slave nodes, send the log shard data to the slave nodes in batches, and after verifying the data integrity at the slave nodes, the slave nodes return confirmation information to the master node; Step S502: Establish a message data upload task queue, scan the confirmed replicated data area in the log shard at a preset time interval, group and package the messages in the data area according to the time range, calculate the storage path of the data packet, asynchronously upload the data packet to the storage bucket corresponding to the main storage area through the object storage interface, and update the storage location index of the data packet.

[0083] Optionally, in this embodiment, a dynamic management strategy is adopted for solid-state drive space allocation. Continuous physical blocks are allocated through the direct I / O interface of the file system to avoid the additional overhead brought by the file system cache. The storage space is pre-allocated in a fixed size, with the initial size of each shard being 1GB and supporting dynamic expansion. The allocation process uses an asynchronous mode, and multiple shards can be created in parallel to improve the resource initialization efficiency.

[0084] This embodiment realizes optimized storage parameter configuration. The storage buffer uses direct memory allocation, bypassing the JVM heap memory management. Each shard is equipped with an independent write buffer and prefetch buffer. The buffer size is dynamically adjusted according to the historical I / O mode. The write buffer is default set to 64MB, and the prefetch buffer size is optimized based on the sequential read mode. The disk flushing policy combines time threshold and data volume threshold, and triggers the disk flushing operation when the accumulated data exceeds the threshold or reaches the time interval.

[0085] This embodiment designs a reliable replication channel mechanism. The replication channel is implemented based on the Netty framework and adopts the zero-copy transmission technology. The channel establishment process includes identity authentication and capability negotiation, and supports SSL encryption and compressed transmission. The data transmission adopts a pipeline design, allowing multiple batches to be sent simultaneously, and the size of each batch is dynamically adjusted according to the network conditions. The sending rate is controlled by a sliding window to avoid network congestion.

[0086] This embodiment constructs a complete data verification process. After receiving the data, the slave node first verifies the integrity of the data packet, including checksum verification and sequence number check. After passing the verification, the data is written to the local storage, and a dual-write mechanism is adopted to ensure data reliability. The confirmation message includes the batch ID and the write location information, and is returned to the master node asynchronously to reduce the confirmation delay.

[0087] This embodiment realizes efficient upload task management. The task queue adopts a priority queue structure, and the priority is calculated based on the time sensitivity of the data and the storage pressure. The queue processing adopts a multi-threaded model, and the number of processing threads is dynamically adjusted according to the number of CPU cores and the IO load. The failure retry mechanism is implemented during the task execution process, and the retry interval adopts an exponential backoff strategy.

[0088] This embodiment designs an innovative data packaging strategy. The grouping process considers multiple dimensions: time range, data volume, access mode, etc. The optimal packaging size is determined through aggregation analysis to balance storage efficiency and retrieval performance. The packaging process supports parallel processing, and data in multiple time ranges can be packaged simultaneously. At the same time, an incremental packaging mechanism is implemented, which only processes newly added data areas.

[0089] This embodiment constructs an intelligent storage path calculation scheme. The path generation considers the time attribute and business attribute of the data and adopts a multi-level directory structure. The path calculation adopts a consistent hashing algorithm to ensure that related data is stored in adjacent positions and improve the batch reading efficiency. At the same time, the path conflict handling is implemented, and the same-name conflict is resolved by appending a unique identifier.

[0090] This embodiment realizes a reliable asynchronous upload mechanism. The object storage interface encapsulates the retry and concurrency control logic and supports segmented upload and resume from breakpoint. The upload process adopts a streaming processing method to improve the transmission efficiency through pipelining design. At the same time, the upload progress tracking is implemented, which supports fine-grained progress query and cancellation operations. The selection of the storage bucket is based on the data life cycle policy, and it supports the automatic migration of data between different storage levels.

[0091] Through the above technical innovations, this embodiment effectively solves the key problems in the distributed storage system: storage performance bottleneck, data replication delay, low upload efficiency, etc. In practical applications, this solution can support large-scale data storage and fast replication requirements. Through multi-level optimization strategies, it significantly improves the throughput and reliability of the system. It is particularly suitable for scenarios that require high-performance storage and disaster recovery, such as financial transactions, log archiving, etc. Through optimized storage strategies and reliable replication mechanisms, the overall performance of the storage system is improved. The systematicness and innovativeness of this solution enable it to adapt to the application requirements of enterprises of different scales. Through continuous optimization and improvement, the comprehensive upgrade of the storage system is realized.

[0092] In an embodiment of the data processing method for kafka storage and computing separation in this application, the following content may also be specifically included: Step S601: Allocate memory space for the message buffer, construct a cache index table to record the mapping relationship between message positions and storage addresses, set the capacity limit and eviction policy of the buffer, establish a hot message scoring mechanism, calculate the message heat score based on the access frequency and time decay factor of the message, and load the message data with a heat score exceeding the preset threshold into the buffer; Step S602: Monitor the read and write load data of the message partition. The load predictor trains a prediction model based on historical load data, calculates the load trend within the future time window, triggers partition expansion when the predicted load exceeds the preset threshold, redistributes the message data in the existing partition to the newly added partition according to the consistent hashing algorithm, and updates the partition routing table.

[0093] Optionally, this embodiment adopts a multi-level cache architecture for memory space allocation. Manage the message buffer through an off-heap memory pool to avoid GC pressure. The memory pool adopts a segmented lock design, divides the overall memory space into multiple regions, and each region is controlled by an independent lock to reduce thread contention. The allocation granularity of the buffer is dynamically adjusted according to the message size, and the buddy algorithm is used for memory block management to effectively reduce memory fragmentation.

[0094] This embodiment implements an efficient cache index structure. Adopt a three-level index design: the first level is the mapping from the position to the partition, and a skip list is used to achieve fast lookup; the second level is the message index within the partition, and a B+ tree structure is adopted; the third level is the memory address mapping of the specific message, and a direct addressing table is used. This multi-level index structure can achieve an O(logn) lookup efficiency in a large number of messages and support range query operations at the same time.

[0095] This embodiment designs an innovative message heat scoring mechanism. The heat score calculation formula is: Score = F×e^(-λt)×W, where F is the access frequency, λ is the time decay coefficient, t is the time interval since the last access, and W is the message weight. The weight factor takes into account the business priority and data size of the message. The scoring process uses a sliding window statistic, and the window size can be dynamically adjusted according to the business characteristics to accurately reflect the real-time heat change of the message.

[0096] This embodiment constructs an intelligent cache eviction policy. Combine the LRU-K and ARC algorithms to implement an adaptive eviction mechanism. Maintain two cache queues: the frequently accessed queue and the generally accessed queue. Messages enter the general queue when they are first accessed, and are upgraded to the frequent queue when the access times exceed the threshold. The size ratio of the two queues is dynamically adjusted according to the access pattern to achieve the optimal utilization of the cache space.

[0097] This embodiment implements an accurate load monitoring system. It collects multi-dimensional load metrics: message throughput, latency distribution, queue depth, etc. Data collection adopts an adaptive sampling strategy, increasing the sampling frequency when the load changes significantly. Historical load data is stored in a time series database to establish a multi-dimensional load profile, providing training data for the prediction model.

[0098] This embodiment designs an advanced load prediction model. It uses an LSTM neural network structure, and the input features include historical load metrics, time features (hours, weeks, holidays), and system events. The model captures the correlations between different features through a multi-head attention mechanism to accurately predict short-term and medium-term load trends. The training process adopts an online learning method to continuously optimize the model parameters to adapt to the dynamic changes of the load pattern.

[0099] This embodiment constructs a reliable partition expansion mechanism. The expansion trigger conditions comprehensively consider multiple factors: predicted load, resource utilization, performance metrics, etc. The expansion process adopts a progressive strategy. First, new partitions are created and preheated, and then message data is reallocated through the consistent hashing algorithm. The hashing algorithm uses virtual node technology to ensure the balance of data distribution while minimizing the amount of data migration.

[0100] This embodiment implements an efficient data migration process. The migration process adopts a batch processing mode, and a migration task queue is set up to manage migration requests. The migration efficiency is improved through multi-threaded parallel processing, and each thread is responsible for migrating a batch of data. At the same time, a migration progress tracking and fault recovery mechanism is implemented to ensure the reliability of the migration process.

[0101] Through the above technological innovations, this embodiment effectively solves the key problems in the distributed message system: low cache hit rate, inaccurate load prediction, poor expansion efficiency, etc. In practical applications, this solution can intelligently identify hot messages and optimize the cache strategy. Through accurate load prediction and efficient expansion mechanism, it significantly improves the system's response speed and scalability. It is particularly suitable for scenarios with obvious access hotspots, such as social media push, real-time recommendation, etc. Through multi-level optimization strategies, the overall performance of the message system is improved. The systematicness and innovativeness of this solution enable it to adapt to the application requirements of enterprises of different scales, and through continuous optimization and improvement, the comprehensive upgrade of the message processing system is achieved.

[0102] In an embodiment of the kafka storage-computation separation data processing method of this application, it may also specifically include the following content: Step S701: The consumer sends a site query request to the coordinator. The coordinator returns the storage location information corresponding to the message site. The consumer searches for the target message data in the message buffer according to the storage location information. If the target message data does not exist in the buffer, it obtains the corresponding data packet from the main storage area, reads the metadata header in the data packet to parse the compression algorithm identifier, and calls the corresponding decompression method to restore the message data; Step S702: Calculate the checksum of the decompressed message data and compare it with the checksum recorded in the metadata header. After passing the check, return the message data to the consumer. After the consumer processes the message, it submits the consumption site to the coordinator. The coordinator records the consumption site in the site management table and periodically persists the site management table to the storage system.

[0103] Optionally, in this embodiment, a multi-level cache mechanism is adopted for site query processing. The coordinator maintains an in-memory cache of site mappings and uses ConcurrentHashMap to achieve fast lookup. The cache is organized by consumer group and partition and uses a two-layer hash table structure. For hot query site information, the access pressure on the coordinator is reduced through local caching. The consistency of the cache is maintained through a version number mechanism. When the site is updated, each consumer is notified to update the local cache through a broadcast mechanism.

[0104] This embodiment realizes intelligent storage location parsing. The storage location information adopts a compressed encoding format, including information such as storage level, file path, offset, etc. The parsing process uses a state machine design and can handle different versions of the encoding format. Through the locality principle of location information, adjacent message locations that may be accessed are predicted and prefetched, reducing subsequent query latency.

[0105] This embodiment designs an efficient cache lookup strategy. When searching in the message buffer, a multi-level index is adopted. First, a Bloom filter is used to quickly determine whether the message may exist, and then a skiplist is used to locate the specific location. When the cache is not hit, asynchronous preloading is started, and subsequent messages that may be accessed are predicted and loaded according to the access pattern of the message, improving the cache hit rate.

[0106] This embodiment constructs a reliable data packet reading mechanism. When obtaining a data packet from the main storage area, an asynchronous IO method is adopted, and the concurrent reading quantity is controlled through a thread pool. The reading of the data packet supports range requests and only obtains the necessary data part. For large data packets, a sharding reading strategy is implemented to avoid loading too much data into memory at one time.

[0107] This embodiment implements an optimized decompression processing flow. The selection of the decompression algorithm is quickly located through the identifier in the metadata header, supporting multiple compression algorithms such as LZ4, Snappy, ZSTD, etc. The decompression process is managed by a thread pool. For a large number of small messages, a batch decompression strategy is adopted to improve efficiency. At the same time, a decompression cache is implemented, and the same compressed data block is decompressed only once.

[0108] This embodiment designs a strict data verification process. The calculation of the verification code uses the hardware-accelerated CRC32 algorithm, and the calculation speed is improved through SIMD instructions. The verification process covers the entire content of the message, including metadata and payload data. When the verification fails, a multi-level retry mechanism is implemented. First, it tries to reread the data. If it continues to fail, it obtains the data from the replica node.

[0109] This embodiment constructs a reliable site submission mechanism. Consumers submit sites asynchronously, and batch submit after accumulating a certain number of sites through the local buffer. The submission process implements idempotency, and repeated submission of the same site will not cause errors. The site submission adopts a two-phase submission protocol to ensure consistency in a distributed environment.

[0110] This embodiment implements an efficient site management strategy. The site management table adopts a segmented lock design to reduce the lock contention for concurrent updates. The table records detailed site information, including consumer group ID, topic, partition, site value, timestamp, etc. The persistence process adopts an incremental method, only saving the changed site information. The reliability of the site data is ensured through the write-ahead log mechanism.

[0111] Through the above technological innovations, this embodiment effectively solves the key problems in distributed message consumption: high site query latency, low data decompression efficiency, unreliable site management, etc. In practical applications, this solution can support large-scale message consumption scenarios. Through multi-level caching and optimized data processing flows, it significantly improves the efficiency and reliability of message consumption. It is particularly suitable for scenarios requiring high throughput and low latency, such as real-time data processing, event stream processing, etc. Through multi-level optimization strategies, the overall performance of the message consumption system is improved. The systematicness and innovativeness of this solution enable it to adapt to the application requirements of enterprises of different scales. Through continuous optimization and improvement, the comprehensive upgrade of the message consumption system is realized.

[0112] In order to effectively solve the deficiencies of traditional technologies in aspects such as memory-computation coupling, storage efficiency, and load balancing, and significantly improve the performance and scalability of the message queue system, this application provides an embodiment of a kafka memory-computation separation data processing device for implementing all or part of the content of the data processing method for kafka memory-computation separation, see Figure 2 The kafka memory-computation separation data processing device specifically includes the following contents: The architecture building module 10 is used to build a multi-layer data processing architecture. The multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing and scheduling layer, and a storage layer. The storage layer includes a stream storage repository. The stream storage repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing and scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate. The message storage module 20 is used to receive batch message data sent by a producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a check code, write the compressed message data into the log shards in the write-ahead log area. The log shards are stored using solid-state drives. A master-slave replication relationship is established between multiple log shards according to the configuration of the replica manager, and the message data in the log shards is asynchronously uploaded to the main storage area. The message cache module 30 is used to create a message cache area. The message cache area is used to store hot message data. The number of message partitions is adjusted according to the prediction result of the load predictor. Consumers read the hot message data from the message cache area based on the message position. When the target message data does not exist in the message cache area, the consumers obtain cold data from the main storage area according to the message position and decompress it. The coordinator records the consumption progress of the consumers and verifies the data integrity based on the check code.

[0113] As can be seen from the above description, the Kafka storage-computation separation data processing device provided by the embodiments of the present application can build a multi-layer data processing architecture, including four layers: network communication, task distribution, processing and scheduling, and storage. The separation of hot and cold data is realized through the hierarchical design of the write-ahead log area and the main storage area. The intelligent scheduling of system resources is realized by combining the replica manager, the coordinator, and the load predictor. The adaptive compression algorithm and the check mechanism are used to ensure the data transmission efficiency and integrity. A message cache area is created to store hot data, and flexible access to hot and cold data is realized based on the message position. This method effectively solves the deficiencies of traditional technologies in aspects such as storage-computation coupling, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system.

[0114] From a hardware perspective, in order to effectively address the deficiencies of traditional technologies in aspects such as memory-computation coupling, storage efficiency, and load balancing, and significantly improve the performance and scalability of the message queue system, this application provides an embodiment of an electronic device for implementing all or part of the content in the data processing method for separating Kafka memory and computation. The electronic device specifically includes the following: A processor, a memory, a communications interface, and a bus; wherein, the processor, the memory, and the communications interface complete communication with each other through the bus; the communications interface is used to implement information transmission between the data processing device for separating Kafka memory and computation and related devices such as the core business system, the user terminal, and the relevant database, etc. This logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., and this embodiment is not limited thereto. In this embodiment, this logic controller can be implemented with reference to the embodiments of the data processing method for separating Kafka memory and computation in the embodiments, as well as the embodiments of the data processing device for separating Kafka memory and computation, and its content is incorporated herein, and the repeated parts will not be elaborated again.

[0115] It can be understood that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0116] In practical applications, part of the data processing method for separating Kafka memory and computation can be executed on the electronic device side as described above, or all operations can be completed in the client device. Specifically, it can be selected according to the processing capacity of the client device and the limitations of the user usage scenario, etc. This application does not make a limitation in this regard. If all operations are completed in the client device, the client device may further include a processor.

[0117] The above-mentioned client device may have a communication module (i.e., a communication unit), and can be communicatively connected to a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, it may also include a server on the intermediate platform, such as a server on a third-party server platform communicatively linked to the task scheduling center server. The server may include a single computer device, or may include a server cluster composed of multiple servers, or a server structure of a distributed device.

[0118] Figure 3 This is a schematic block diagram of the system composition of the electronic device 9600 according to the embodiment of this application. As Figure 3As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It should be noted that this Figure 3 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunications functions or other functions.

[0119] In one embodiment, the data processing method function of separating storage and computing in Kafka can be integrated into the central processing unit 9100. Among them, the central processing unit 9100 can be configured to perform the following controls: Step S101: Build a multi-layer data processing architecture. The multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing scheduling layer, and a storage layer. The storage layer includes a stream repository. The stream repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate; Step S102: Receive the batch message data sent by the producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a check code, write the compressed message data into the log shards in the write-ahead log area. The log shards are stored using solid-state drives. Establish a master-slave replication relationship among multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shards to the main storage area; Step S103: Create a message buffer area. The message buffer area is used to store hot message data. Adjust the number of message partitions according to the prediction result of the load predictor. The consumer reads the hot message data from the message buffer area based on the message offset. When the target message data does not exist in the message buffer area, the consumer obtains the cold data from the main storage area according to the message offset and decompresses it. The coordinator records the consumption progress of the consumer and verifies the data integrity based on the check code.

[0120] As can be seen from the above description, the electronic device provided by the embodiments of the present application constructs a multi-layer data processing architecture, including four levels: network communication, task distribution, processing scheduling, and storage. Through the hierarchical design of the write-ahead log area and the main storage area, hot and cold data separation is achieved. Combining the replica manager, coordinator, and load predictor realizes the intelligent scheduling of system resources. The adaptive compression algorithm and verification mechanism are adopted to ensure the data transmission efficiency and integrity. A message buffer is created to store hot data, and flexible access to hot and cold data is realized based on message positions. This method effectively solves the deficiencies of traditional technologies in aspects such as memory-computation coupling, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system.

[0121] In another embodiment, the data processing device for separating memory and computing of Kafka can be separately configured from the central processing unit 9100. For example, the data processing device for separating memory and computing of Kafka can be configured as a chip connected to the central processing unit 9100, and the functions of the data processing method for separating memory and computing of Kafka are realized through the control of the central processing unit.

[0122] As Figure 3 shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It should be noted that the electronic device 9600 does not necessarily have to include Figure 3 all the components shown in Figure 3 ; in addition, the electronic device 9600 may further include

[0123] components not shown in Figure 3 ; reference may be made to the prior art.

[0124] Among them, the memory 9140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices. The above information related to failures can be stored, and in addition, programs for executing relevant information can be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0125] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to supply power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0126] The memory 9140 may be a solid-state memory, for example, a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that stores information even when power is off, can be selectively erased and has more data. Examples of such a memory are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or the processes for operating the electronic device 9600 by the central processing unit 9100.

[0127] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers of the electronic device for communication functions and / or for performing other functions of the electronic device (such as a messaging application, an address book application, etc.).

[0128] The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which may be the same as in the case of a conventional mobile communication terminal.

[0129] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module, etc. The communication module 9110 (transmitter / receiver) is also coupled to the speaker 9131 and the microphone 9132 via the audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby implementing normal telecommunication functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is also coupled to the central processing unit 9100, so that recording can be performed on the local device via the microphone 9132 and the sound stored on the local device can be played via the speaker 9131.

[0130] An embodiment of the present application further provides a computer-readable storage medium that can implement all steps of the kafka storage-computation separation data processing method where the execution subject in the above embodiment is a server or a client. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements all steps of the kafka storage-computation separation data processing method where the execution subject in the above embodiment is a server or a client. For example, when the processor executes the computer program, the following steps are implemented: Step S101: Build a multi-layer data processing architecture. The multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing scheduling layer, and a storage layer. The storage layer includes a stream storage repository. The stream storage repository includes a write-ahead log area and a main storage area. Divide multiple log shards in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate. Step S102: Receive batch message data sent by a producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a check code, write the compressed message data into the log shards in the write-ahead log area. The log shards are stored using solid-state drives. Establish a master-slave replication relationship among multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shards to the main storage area. Step S103: Create a message buffer area. The message buffer area is used to store hot message data. Adjust the number of message partitions according to the prediction result of the load predictor. A consumer reads the hot message data from the message buffer area based on the message offset. When the target message data does not exist in the message buffer area, the consumer obtains cold data from the main storage area according to the message offset and decompresses it. The coordinator records the consumption progress of the consumer and verifies the data integrity based on the check code.

[0131] As can be seen from the above description, the computer-readable storage medium provided by the embodiments of the present application constructs a multi-layer data processing architecture, including four layers: network communication, task distribution, processing scheduling, and storage. Through the hierarchical design of the write-ahead log area and the main storage area, hot and cold data separation is achieved. Combining the replica manager, coordinator, and load predictor enables intelligent scheduling of system resources. The adaptive compression algorithm and verification mechanism are used to ensure data transmission efficiency and integrity. A message buffer is created to store hot data, and flexible access to hot and cold data is realized based on message positions. This method effectively solves the deficiencies of traditional technologies in terms of memory-computation coupling, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system.

[0132] An embodiment of the present application also provides a computer program product that can implement all the steps in the data processing method for separating memory and computing of Kafka where the execution subject in the above embodiments is a server or a client. When the computer program / instructions are executed by a processor, the steps of the data processing method for separating memory and computing of Kafka are implemented. For example, the computer program / instructions implement the following steps: Step S101: Construct a multi-layer data processing architecture. The multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing scheduling layer, and a storage layer. The storage layer includes a stream storage repository. The stream storage repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate. Step S102: Receive batch message data sent by a producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a check code, write the compressed message data into a log shard in the write-ahead log area. The log shard is stored using a solid-state drive. Establish a master-slave replication relationship among multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shard to the main storage area. Step S103: Create a message buffer. The message buffer is used to store hot message data. Adjust the number of message partitions according to the prediction result of the load predictor. Consumers read the hot message data from the message buffer based on message positions. When the target message data does not exist in the message buffer, the consumers obtain cold data from the main storage area according to message positions and decompress it. The coordinator records the consumption progress of the consumers and verifies the data integrity based on the check code.

[0133] As can be seen from the above description, the computer program product provided by the embodiments of the present application constructs a multi-layer data processing architecture, including four layers: network communication, task distribution, processing scheduling, and storage. The separation of hot and cold data is achieved through the hierarchical design of the write-ahead log area and the main storage area. The intelligent scheduling of system resources is realized by combining the replica manager, coordinator, and load predictor. The adaptive compression algorithm and verification mechanism are adopted to ensure the data transmission efficiency and integrity. A message buffer is created to store hot data, and flexible access to hot and cold data is achieved based on message positions. This method effectively solves the deficiencies of traditional technologies in terms of memory-computation coupling, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system.

[0134] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, apparatus, or computer program product. Therefore, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0136] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are performed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide for implementing the specified functions in the flowFigure 1 One process or multiple processes and / or boxes Figure 1 Steps of functions specified in one box or multiple boxes.

[0138] In the present invention, specific embodiments are used to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A data processing method for Kafka storage and computing separation, characterized in that: The method comprises: Construct a multi-layer data processing architecture, the multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing scheduling layer and a storage layer, the storage layer includes a stream storage library, the stream storage library includes a pre-write log area and a main storage area, a plurality of log shards are divided in the pre-write log area, the main storage area realizes data persistence through an object storage service, the processing scheduling layer includes a replica manager, a coordinator and a load predictor, the replica manager is responsible for data replica synchronization, the coordinator is responsible for partition allocation and group management, and the load predictor establishes a prediction model based on system resource utilization; Receive batch message data sent by the producer, calculate the data features of the batch message data, select a compression algorithm according to the data features, perform compression encoding on the batch message data to generate a checksum, write the compressed message data into the log shards in the pre-write log area, the log shards are stored in solid-state hard disks, establish a master-slave replication relationship between the multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shards to the main storage area; A message buffer is created, where the message buffer is used to store hot message data. The number of message partitions is adjusted according to the prediction result of the load predictor. The consumer reads the hot message data from the message buffer based on the message site. When the target message data does not exist in the message buffer, the consumer obtains and decompresses the cold data from the main storage area according to the message site. The coordinator records the consumer's consumption progress and verifies the data integrity based on the check code.

2. The data processing method for separating Kafka storage and computing according to claim 1 is characterized in that: The multi-layer data processing architecture is constructed, and the multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing scheduling layer and a storage layer. The storage layer includes a stream storage library, and the stream storage library includes a pre-write log area and a main storage area, including: Divide the data processing architecture into call links, set up socket listeners in the network communication layer to handle client connection requests, create message transmission channels and configure transmission parameters, build routing tables in the task distribution layer to map the correspondence between processors and message types, establish task queues, set up task priority scheduling mechanisms, and distribute tasks to corresponding processors according to message types; Deploy a replica manager at the processing and scheduling layer to implement partition master-slave replication, configure a coordinator to manage partition allocation and consumer groups, deploy a load predictor to collect system resource usage data, create a stream repository at the storage layer and divide the pre-write log area and the main storage area. The pre-write log area uses a solid-state hard disk storage medium, and the main storage area uses an object storage service as persistent storage.

3. The data processing method for Kafka storage and computing separation according to claim 1 is characterized in that: The write-ahead log area is divided into a plurality of log shards, the main storage area realizes data persistence through an object storage service, the processing scheduling layer includes a replica manager, a coordinator and a load predictor, the replica manager is responsible for data replica synchronization, the coordinator is responsible for partition allocation and group management, and the load predictor establishes a prediction model based on system resource utilization, including: The number of log shards is calculated based on the total capacity of the pre-write log area, a unique identifier is assigned to each log shard, the log shards are organized into a linked list structure in chronological order, a message offset and a timestamp index are set in the log shards, a reserved space threshold of the log shards is configured, a mapping relationship between the log shards and the physical storage device is established, the main storage area is divided into buckets according to the time range, and a metadata index is created for each bucket; Configure the master-slave replication parameters of the replica manager, establish a synchronization mechanism for partition replicas, build a partition allocation table and a consumer group management table in the coordinator, record consumer group member information and consumption progress, and the load predictor collects processor usage, memory occupancy, disk read and write rate system resource indicators, builds a time series prediction model to calculate resource usage trends, and generates a load balancing strategy based on the prediction results.

4. The data processing method for Kafka storage and computing separation according to claim 1 is characterized in that: The receiving of batch message data sent by the producer, calculating data features of the batch message data, selecting a compression algorithm according to the data features, performing compression encoding on the batch message data to generate a checksum, and writing the compressed message data into a log shard in the write-ahead log area includes: Parse the message header and message body of the batch message data sent by the producer, count the data distribution characteristics of the message body, calculate the entropy value, repetition, and value range data characteristic indicators of the message body, establish a compression algorithm evaluation matrix, calculate the compression rate and processing delay of different compression algorithms based on the data characteristic indicators, and select the optimal compression algorithm in the evaluation matrix; Apply the selected compression algorithm to the batch message data to generate a compressed byte stream, calculate the CRC32 check code of the compressed byte stream, package the compressed data length, compression algorithm identifier, and check code into a metadata header, assemble the metadata header and the compressed byte stream into a data block, and select an available log segment in the pre-write log area to write the data block.

5. The data processing method for Kafka storage and computing separation according to claim 1 is characterized in that: The log shards are stored in solid state hard disks, a master-slave replication relationship is established between the plurality of log shards according to the configuration of the replica manager, and the message data in the log shards are asynchronously uploaded to the main storage area, including: Allocate SSD storage space for log shards, configure storage parameters for log shards, set storage buffer size and flushing strategy, select slave nodes based on the configuration information of the replica manager, establish replication channels between master and slave nodes, send log shard data to slave nodes according to batch size, and return confirmation information to the master node after the slave node verifies data integrity; Establish a message data upload task queue, scan the confirmed replicated data area in the log shard at preset time intervals, group and package the messages in the data area according to the time range, calculate the storage path of the data packet, asynchronously upload the data packet to the storage bucket corresponding to the main storage area through the object storage interface, and update the storage location index of the data packet.

6. The data processing method for Kafka storage and computing separation according to claim 1 is characterized in that: The creating of a message buffer area, wherein the message buffer area is used to store hotspot message data, and adjusting the number of message partitions according to the prediction result of the load predictor, comprises: Allocate memory space for the message cache, build a cache index table to record the mapping relationship between message locations and storage addresses, set the cache capacity limit and elimination strategy, establish a hot message scoring mechanism, calculate the message heat score based on the message access frequency and time decay factor, and load the message data whose heat score exceeds the preset threshold into the cache; Monitor the read and write load data of the message partition. The load predictor trains a prediction model based on historical load data, calculates the load trend in the future time window, triggers partition expansion when the predicted load exceeds a preset threshold, redistributes the message data in the existing partition to the newly added partition according to the consistent hashing algorithm, and updates the partition routing table.

7. The data processing method for Kafka storage and computing separation according to claim 1 is characterized in that: The consumer reads the hot message data from the message buffer area based on the message location. When the target message data does not exist in the message buffer area, the consumer obtains the cold data from the main storage area according to the message location and decompresses it. The coordinator records the consumer's consumption progress and verifies the data integrity based on the check code, including: The consumer sends a location query request to the coordinator, and the coordinator returns the storage location information corresponding to the message location. The consumer searches for the target message data in the message cache according to the storage location information. If the target message data does not exist in the cache, the consumer obtains the corresponding data packet from the main storage area, reads the metadata header in the data packet to parse the compression algorithm identifier, and calls the corresponding decompression method to restore the message data. The checksum of the decompressed message data is calculated and compared with the checksum recorded in the metadata header. After the checksum passes, the message data is returned to the consumer. After processing the message, the consumer submits the consumption location to the coordinator. The coordinator records the consumption location in the location management table and regularly persists the location management table to the storage system.

8. A data processing device with Kafka storage and computing separation, characterized in that: The device comprises: An architecture building module is used to build a multi-layer data processing architecture, wherein the multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing scheduling layer and a storage layer, wherein the storage layer includes a stream storage library, the stream storage library includes a pre-write log area and a main storage area, wherein a plurality of log shards are divided in the pre-write log area, and the main storage area realizes data persistence through an object storage service, wherein the processing scheduling layer includes a replica manager, a coordinator and a load predictor, wherein the replica manager is responsible for data replica synchronization, the coordinator is responsible for partition allocation and group management, and the load predictor establishes a prediction model based on system resource utilization; A message storage module, used to receive batch message data sent by a producer, calculate data features of the batch message data, select a compression algorithm according to the data features, perform compression encoding on the batch message data to generate a checksum, write the compressed message data into a log shard in the pre-write log area, the log shard is stored in a solid-state hard disk, establish a master-slave replication relationship between the plurality of log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shard to the main storage area; A message cache module is used to create a message cache area, wherein the message cache area is used to store hot message data, and the number of message partitions is adjusted according to the prediction result of the load predictor. The consumer reads the hot message data from the message cache area based on the message site. When the target message data does not exist in the message cache area, the consumer obtains and decompresses the cold data from the main storage area according to the message site. The coordinator records the consumer's consumption progress and verifies the data integrity based on the check code.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the data processing method of Kafka storage and computing separation as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data processing method for Kafka storage and computing separation as described in any one of claims 1 to 7 are implemented.

Images