Data Processing Method and Device for Separating Storage and Computing in Kafka

Abstract

An embodiment of the present application provides a data processing method and device for Kafka storage and computing separation. By constructing a multi-layer data processing architecture, including four levels: network communication, task distribution, processing scheduling, and storage. Through the hierarchical design of the write-ahead log area and the main storage area, hot and cold data separation is achieved. Combining the replica manager, coordinator, and load predictor to realize the intelligent scheduling of system resources. The adaptive compression algorithm and verification mechanism are adopted to ensure the data transmission efficiency and integrity. A message buffer is created to store hot data, and flexible access to hot and cold data is realized based on the message position. This method effectively solves the deficiencies of traditional technologies in aspects such as storage and computing coupling, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system.

Description

Technical Field

[0001] This application relates to the field of data processing, and particularly to a data processing method and device for Kafka storage-computation separation. Background Art

[0002] Existing Kafka data processing methods have obvious deficiencies. In the traditional architecture, the coupling degree between storage and computing is high, making it difficult to achieve elastic expansion of resources. At the same time, there is a problem of uneven data access hotspots.

[0003] In addition, there are bottlenecks in the data storage efficiency of existing technologies. Most systems fail to effectively distinguish between hot and cold data and lack intelligent data compression and caching mechanisms, resulting in low storage space utilization and large access latency.

[0004] Existing systems have technical shortcomings in load balancing and fault tolerance mechanisms. They lack dynamic partition adjustment and prediction capabilities and are difficult to cope with sudden traffic and system failures. Solving these problems is of great significance for improving the performance and reliability of the message queue system. Summary of the Invention

[0005] In view of the problems in the prior art, this application provides a data processing method and device for Kafka storage-computation separation, which can effectively solve the deficiencies of traditional technologies in aspects such as storage-computation coupling, storage efficiency, and load balancing, and significantly improve the performance and scalability of the message queue system.

[0006] To solve at least one of the above problems, this application provides the following technical solutions:

[0007] In a first aspect, this application provides a data processing method for Kafka storage-computation separation, including:

[0008] Construct a multi-layer data processing architecture, the multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing and scheduling layer, and a storage layer. The storage layer includes a stream storage repository, the stream storage repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area, and the main storage area realizes data persistence through an object storage service. The processing and scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization, the coordinator is responsible for partition allocation and group management, and the load predictor establishes a prediction model based on the system resource utilization rate;

[0009] Receive the batch message data sent by the producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a check code, write the compressed message data into the log shard in the write-ahead log area, where the log shard is stored using a solid-state drive, establish a master-slave replication relationship among multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shard to the main storage area;

[0010] Create a message buffer area for storing hot message data, adjust the number of message partitions according to the prediction result of the load predictor, and the consumer reads the hot message data from the message buffer area based on the message position. When the target message data does not exist in the message buffer area, the consumer obtains the cold data from the main storage area according to the message position and decompresses it. The coordinator records the consumption progress of the consumer and verifies the data integrity based on the check code.

[0011] Further, it also includes: dividing the data processing architecture into call chains, setting up socket listeners in the network communication layer to handle client connection requests, creating a message transmission channel and configuring transmission parameters, building a routing table in the task distribution layer to map the corresponding relationship between processors and message types, establishing a task queue, setting up a task priority scheduling mechanism, and distributing tasks to the corresponding processors according to the message type;

[0012] Deploy a replica manager in the processing and scheduling layer to implement partition master-slave replication, configure a coordinator to manage partition allocation and consumer groups, deploy a load predictor to collect system resource usage data, create a stream repository in the storage layer and divide it into a write-ahead log area and a main storage area. The write-ahead log area uses a solid-state drive storage medium, and the main storage area uses an object storage service as persistent storage.

[0013] Further, it also includes: calculating the number of log shards based on the total capacity of the write-ahead log area, assigning a unique identifier to each log shard, organizing the log shards into a linked list structure in chronological order, setting message offsets and timestamp indexes in the log shards, configuring the reserved space threshold of the log shards, and establishing a mapping relationship between the log shards and physical storage devices. The main storage area is divided into storage buckets according to time ranges, and metadata indexes are created for each storage bucket;

[0014] Configure the master-slave replication parameters of the replica manager, establish a synchronization mechanism for partition replicas, build a partition allocation table and a consumer group management table in the coordinator, record consumer group member information and consumption progress. The load predictor collects system resource metrics such as processor utilization rate, memory occupancy rate, and disk read / write rate, constructs a time series prediction model to calculate resource usage trends, and generates a load balancing strategy based on the prediction results.

[0015] Furthermore, it also includes: parsing the message headers and message bodies of the batch message data sent by producers, statistically analyzing the data distribution characteristics of the message bodies, calculating data characteristic metrics such as the entropy value, repetition degree, and numerical range of the message bodies, establishing a compression algorithm evaluation matrix, calculating the compression ratios and processing delays of different compression algorithms based on the data characteristic metrics, and selecting the optimal compression algorithm in the evaluation matrix;

[0016] Applying the selected compression algorithm to the batch message data to generate a compressed byte stream, calculating the CRC32 checksum of the compressed byte stream, packing the compressed data length, compression algorithm identifier, and checksum into a metadata header, assembling the metadata header and the compressed byte stream into a data block, and selecting an available log shard in the write-ahead log area to write the data block.

[0017] Furthermore, it also includes: allocating solid-state drive storage space for the log shards, configuring the storage parameters of the log shards, setting the storage buffer size and disk flushing policy, selecting slave nodes based on the configuration information of the replica manager, establishing a replication channel between the master and slave nodes, sending the log shard data to the slave nodes in batches, and having the slave nodes return confirmation information to the master node after verifying the data integrity;

[0018] Establishing a message data upload task queue, scanning the confirmed replicated data areas in the log shards at a preset time interval, grouping and packing the messages in the data areas according to a time range, calculating the storage paths of the data packets, asynchronously uploading the data packets to the storage buckets corresponding to the main storage area through the object storage interface, and updating the storage location index of the data packets.

[0019] Furthermore, it also includes: allocating memory space for the message buffer, constructing a cache index table to record the mapping relationship between message positions and storage addresses, setting the capacity limit and eviction policy of the buffer, establishing a hot message scoring mechanism, calculating the message heat score based on the access frequency of the messages and the time decay factor, and loading the message data with a heat score exceeding the preset threshold into the buffer;

[0020] Monitoring the read and write load data of the message partitions, training a prediction model by the load predictor based on historical load data, calculating the load trend within a future time window, triggering partition expansion when the predicted load exceeds the preset threshold, redistributing the message data in the existing partitions to the newly added partitions according to the consistent hashing algorithm, and updating the partition routing table.

[0021] Further, it further includes: The consumer sends a site query request to the coordinator, the coordinator returns the storage location information corresponding to the message site, the consumer searches for the target message data in the message buffer according to the storage location information. If the target message data does not exist in the buffer, it obtains the corresponding data packet from the main storage area, reads the metadata header in the data packet to parse the compression algorithm identifier, and calls the corresponding decompression method to restore the message data;

[0022] Calculate the checksum of the decompressed message data and compare it with the checksum recorded in the metadata header. After the verification passes, return the message data to the consumer. After the consumer processes the message, it submits the consumption site to the coordinator, and the coordinator records the consumption site in the site management table and periodically persists the site management table to the storage system.

[0023] In a second aspect, the present application provides a data processing device for Kafka storage and computing separation, including:

[0024] An architecture building module, used to build a multi-layer data processing architecture. The multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing scheduling layer, and a storage layer. The storage layer includes a stream storage repository. The stream storage repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor builds a prediction model based on the system resource utilization rate;

[0025] A message storage module, used to receive batch message data sent by the producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a checksum, write the compressed message data into the log shard in the write-ahead log area. The log shard is stored using a solid-state drive, establish a master-slave replication relationship between multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shard to the main storage area;

[0026] A message cache module, used to create a message buffer. The message buffer is used to store hot message data, adjust the number of message partitions according to the prediction result of the load predictor. The consumer reads the hot message data from the message buffer based on the message site. When the target message data does not exist in the message buffer, the consumer obtains cold data from the main storage area according to the message site and decompresses it. The coordinator records the consumption progress of the consumer and verifies the data integrity based on the checksum.

[0027] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the data processing method for separating Kafka storage and computing are implemented.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method for separating Kafka storage and computing are implemented.

[0029] In a fifth aspect, the present application provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the data processing method for separating Kafka storage and computing are implemented.

[0030] As can be seen from the above technical solutions, the present application provides a data processing method and device for separating Kafka storage and computing. By constructing a multi-layer data processing architecture, including four layers: network communication, task distribution, processing scheduling, and storage. The separation of hot and cold data is achieved through the hierarchical design of the write-ahead log area and the main storage area, and the intelligent scheduling of system resources is realized by combining the replica manager, the coordinator, and the load predictor. The adaptive compression algorithm and the verification mechanism are used to ensure the data transmission efficiency and integrity. A message buffer is created to store hot data, and flexible access to hot and cold data is realized based on the message offset. This method effectively solves the deficiencies of the traditional technology in aspects such as storage-computation coupling, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0032] Figure 1 It is a schematic flowchart of the data processing method for separating Kafka storage and computing in the embodiments of the present application;

[0033] Figure 2 It is a structural diagram of the data processing device for separating Kafka storage and computing in the embodiments of the present application;

[0034] Figure 3 It is a schematic structural diagram of the electronic device in the embodiments of the present application.

[0035] Reference Signs:

[0036] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed implementation manners

[0037] To make the objectives, technical solutions and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Apparently, the described embodiments are some but not all of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0038] In the technical solutions of this application, the acquisition, storage, use, processing, etc. of data all comply with the relevant provisions of national laws and regulations.

[0039] Considering the problems existing in the prior art, this application provides a data processing method and device for separating storage and computing in Kafka. By constructing a multi-layer data processing architecture, including four layers: network communication, task distribution, processing scheduling, and storage. The separation of hot and cold data is achieved through the hierarchical design of the write-ahead log area and the main storage area. The intelligent scheduling of system resources is realized by combining the replica manager, coordinator, and load predictor. The adaptive compression algorithm and verification mechanism are used to ensure the data transmission efficiency and integrity. A message buffer is created to store hot data, and flexible access to hot and cold data is realized based on message positions. This method effectively solves the deficiencies of traditional technologies in aspects such as storage-computation coupling, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system.

[0040] To effectively solve the deficiencies of traditional technologies in aspects such as storage-computation coupling, storage efficiency, and load balancing, and significantly improve the performance and scalability of the message queue system, this application provides an embodiment of a data processing method for separating storage and computing in Kafka. Refer to Figure 1 The data processing method for separating storage and computing in Kafka specifically includes the following content:

[0041] Step S101: Build a multi-layer data processing architecture. The multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing scheduling layer, and a storage layer. The storage layer includes a stream repository, and the stream repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate.

[0042] Optionally, based on the architecture characteristics of Kafka, this embodiment constructs a data processing architecture with clear hierarchical divisions. In the network communication layer, a high-performance network communication is implemented using an asynchronous IO framework based on Netty. Multiple processors are organized through a Pipeline mechanism, including codec, idle detector, and compression processors. Each processor independently completes a specific function, and the processors cooperate through an event passing mechanism to achieve high-concurrency processing capabilities at the network level.

[0043] This embodiment implements an intelligent routing mechanism in the task distribution layer. A message partition routing table is constructed through a consistent hashing algorithm to ensure uniform distribution of messages among partitions. The routing table adopts a two-layer structure. The first layer is the mapping from the topic to the partition, and the second layer is the mapping from the partition to the processor. When partition expansion or contraction is required, through the rebalancing operation of the consistent hashing ring, the affected messages are re-routed to minimize the data migration cost.

[0044] This embodiment designs an innovative processing scheduling layer structure. The replica manager adopts a consistent replication mechanism based on the Raft protocol to ensure data consistency through a write-ahead log (WAL). Each partition maintains an independent replication group, and the members within the group detect the node status through a heartbeat mechanism. When the primary node fails, a new primary node is elected through an election algorithm. The coordinator is responsible for the management of the consumer group and implements a consumer balancing algorithm to ensure balanced distribution of partition loads to consumers.

[0045] This embodiment implements a deep learning model based on LSTM (Long Short-Term Memory) in the load predictor. The model receives multi-dimensional time series data as input, including indicators such as CPU utilization rate, memory occupancy rate, disk IO rate, and network bandwidth. The time-dependent relationships of these indicators are captured through a multi-layer LSTM network to predict the resource usage trend within a future time window. The prediction results are used to guide partition scaling decisions to effectively prevent system overload.

[0046] In this embodiment, a two - layer storage structure is created in the storage layer. The write - ahead log area uses solid - state drives for storage and achieves high - speed writing through zero - copy technology. The size of the log shard is dynamically adjusted according to the expected throughput. Each shard contains an index file and a data file. The index file records the mapping from the message offset to the physical location, supporting fast message location. The data file uses the append - write method to ensure sequential I / O performance. The main storage area realizes data persistence through the object storage service and adopts a hierarchical storage strategy, preferentially storing hot data in the storage layer with higher performance.

[0047] This embodiment implements an efficient data synchronization mechanism. The main storage area adopts an asynchronous upload strategy, sets up a data upload task queue, and batches the data in the write - ahead log area and uploads it to the object storage at predefined time intervals. During the upload process, a multi - level caching strategy is used to cache frequently accessed data blocks in memory, reducing the access frequency to the object storage. At the same time, an intelligent data pre - warming mechanism is implemented, predicting the data blocks that may be accessed according to the access pattern and loading them into the cache in advance.

[0048] This embodiment constructs a complete monitoring and alerting system. Standardized metric collection points are established between layers to monitor the running status of the system in real - time. When abnormal situations are detected, such as network latency exceeding the threshold, insufficient storage space, etc., the operation and maintenance personnel are notified through a multi - level alerting mechanism. At the same time, an automated fault recovery mechanism is implemented, and recovery operations can be automatically executed for common fault scenarios.

[0049] Through the above - mentioned technological innovations, this embodiment effectively solves the key problems existing in the traditional Kafka architecture: poor scalability, insufficient data reliability, low resource utilization, etc. caused by the coupling of storage and computing. In practical applications, this solution can support large - scale distributed message - processing scenarios, significantly improving the scalability and resource utilization of the system through a storage - computing separation architecture. It is particularly suitable for scenarios with high requirements for data reliability and processing performance, such as finance and e - commerce. Through the multi - layer architecture design and intelligent scheduling strategies, the overall performance of the message - processing system is improved.

[0050] Step S102: Receive the batch message data sent by the producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a check code, write the compressed message data into the log shard in the write - ahead log area. The log shard uses solid - state drives for storage, establish a master - slave replication relationship between multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shard to the main storage area;

[0051] Optionally, in this embodiment, a batch processing mechanism is adopted for message reception and processing. The producer organizes messages using a ProducerBatch object and controls the data volume of messages in a single batch through a configurable batch size parameter. The message reception buffer adopts a circular buffer design and controls the writing and extraction of data through read and write pointers. When the data volume in the buffer reaches the batch threshold or the waiting time exceeds the configured delay threshold, the processing flow of batch messages is triggered.

[0052] This embodiment implements an adaptive data feature analysis mechanism. Multi-dimensional feature extraction is performed on batch message data, including repeated pattern analysis, entropy value calculation, and numerical distribution statistics. Repeated patterns are identified through the LZ77 algorithm with a sliding window, and the occurrence frequency of repeated substrings in the data is calculated.

[0053] The entropy value H is calculated by the formula H = -∑(Pi×log2(Pi)), where Pi represents the probability of different byte values occurring. The lower the entropy value, the stronger the compressibility of the data. Numerical distribution is statistically analyzed using a histogram to identify the concentration and dispersion of the data.

[0054] This embodiment designs an intelligent compression algorithm selection strategy. A compression algorithm feature matrix is established, including common compression algorithms such as LZ4, Snappy, and ZSTD. For each algorithm, its compression ratio and compression speed metrics under different data characteristics are recorded. Through a multi-factor decision-making model, the optimal compression algorithm is selected by comprehensively considering data characteristics, algorithm performance, and system resource status. For example, for log data with significant repeated patterns, the LZ4 algorithm is preferentially selected; for time-series data with concentrated numerical distribution, a compression algorithm specifically for numerical values is preferentially selected.

[0055] This embodiment implements an efficient compression processing flow. The compression process adopts multi-threaded parallel processing, and the batch message data is sliced and compressed in parallel. The compression result includes a compressed data block and a metadata header, and the metadata header records information such as the compression algorithm type, the original data size, and the compressed size. At the same time, a CRC32 checksum is generated, and the checksum calculation covers the compressed data and the metadata header to ensure data integrity.

[0056] This embodiment constructs a reliable data writing mechanism. The write-ahead log area is stored on a solid-state drive, and the write performance is improved through asynchronous writing. The writing of log slices adopts an append writing mode, and each slice maintains a write position pointer to ensure sequential writing. The memory mapping file technology is used during the writing process to reduce system call overhead. At the same time, write buffer management is implemented, and when the buffer utilization rate exceeds the threshold, an asynchronous disk flushing operation is triggered.

[0057] This embodiment designs an innovative master-slave replication mechanism. The replica manager maintains data consistency between the master and slave nodes through a Raft-based consensus protocol. After receiving a write request, the master node first writes the data to the local log shard and then sends it to all slave nodes in parallel. The pipeline replication technology is adopted to send the next batch while waiting for the confirmation of the previous batch, improving the replication efficiency. After receiving the data, the slave node verifies the data integrity, writes the data to the local log shard, and returns a confirmation message to the master node.

[0058] This embodiment implements an intelligent data upload strategy. The main storage area adopts an object storage service to support long-term data persistence. The data upload is asynchronous, and an upload task queue is set up to manage the upload requests. The upload frequency is dynamically adjusted according to the usage of the write-ahead log area, and the upload frequency is increased when the log shard utilization rate is high. The upload process supports resume from breakpoint. When the upload is interrupted due to network problems, it can continue to upload from the breakpoint position.

[0059] Through the above technological innovations, this embodiment effectively solves the key problems in message processing: low data compression efficiency, storage performance bottleneck, insufficient data reliability, etc. In practical applications, this solution can adapt to different types of message data. Through an intelligent compression strategy and a reliable storage mechanism, it significantly improves the data processing efficiency and reliability. It is particularly suitable for scenarios such as large-scale log collection and time-series data processing. Through a multi-level storage architecture and an asynchronous processing mechanism, the overall performance of the message processing system is improved. The systematicness and innovativeness of this solution enable it to adapt to the application requirements of enterprises of different scales. Through continuous optimization and improvement, the overall message processing efficiency is improved.

[0060] Step S103: Create a message buffer area for storing hot message data. Adjust the number of message partitions according to the prediction result of the load predictor. The consumer reads the hot message data from the message buffer area based on the message position. When the target message data does not exist in the message buffer area, the consumer obtains and decompresses the cold data from the main storage area according to the message position. The coordinator records the consumption progress of the consumer and verifies the data integrity based on the checksum.

[0061] Optionally, based on the message access characteristics, this embodiment implements a multi-level cache management architecture. The message buffer area adopts an off-heap memory design, directly allocating memory space through DirectByteBuffer, avoiding the limitations of JVM heap memory management. The buffer area is divided into multiple memory pages of a fixed size, and each memory page maintains an independent memory mapping table to record the position information of the message in the physical memory. The zero-copy technology is used to achieve fast data reading, significantly reducing the memory copy overhead.

[0062] This embodiment designs an innovative hot news recognition mechanism. The access frequency of messages is statistically counted using a sliding time window, and the window size can be dynamically adjusted according to business characteristics. A heat score is calculated for each message, and the score calculation formula is:

[0063] Score = Frequency×Recency×Weight,

[0064] where Frequency is the access frequency, Recency is the time decay factor, and Weight is the message priority weight. The natural decay process of message heat over time is simulated through an exponential decay function to accurately reflect the real-time heat change of messages.

[0065] This embodiment implements an intelligent partition dynamic adjustment strategy. The load predictor analyzes historical load data through a deep learning model and establishes a time series prediction network based on LSTM. The input features include indicators such as partition read-write QPS, message backlog, and consumption latency. The time series dependence relationship between these indicators is learned through a multi-layer LSTM network. The model outputs the predicted load trend within a future time window, and when the predicted load exceeds the preset threshold, a partition expansion operation is triggered.

[0066] This embodiment constructs an efficient message reading process. The consumer first obtains the consumption position information from the coordinator. The coordinator maintains a position mapping table for the consumption group, recording the consumption progress of each consumer. The consumer calculates the logical offset of the message based on the position information and quickly locates the position of the message in the buffer through multi-level lookups. If the target message is not in the buffer, the cold data loading process is started to obtain the corresponding compressed data packet from the main storage area.

[0067] This embodiment designs a reliable data integrity verification mechanism. During the message decompression process, first parse the metadata header of the data packet to obtain the compression algorithm type and checksum information. Select the corresponding decompression method according to the compression algorithm type. After decompression, calculate the CRC32 checksum of the message data and compare it with the checksum in the metadata header. If the verification fails, trigger the retry mechanism to obtain the data from the storage system again to ensure data integrity.

[0068] This embodiment implements an optimized cache eviction strategy. An improved LRU-K algorithm is used to manage the cache space, recording the timestamps of the last K accesses of each message, and predicting the future access probability of messages more accurately through these access records. When the cache space is insufficient, messages with a lower access probability are preferentially evicted. At the same time, a preloading mechanism is implemented to predict the messages that may be accessed according to the access pattern of the messages and preload them into the buffer in advance.

[0069] This embodiment constructs a complete consumption progress management mechanism. The coordinator persists the consumption positions to the storage system regularly and uses the two-phase commit protocol to ensure the atomicity of position updates. The position information includes fields such as the consumer group ID, topic partition, consumption position, and timestamp, and a version number mechanism is used to handle concurrent updates. When the consumer restarts or rebalancing occurs, the previous consumption progress can be quickly restored.

[0070] Through the above technological innovations, this embodiment effectively solves the key problems in the distributed message system: high access latency of hot messages, low cache utilization rate, load imbalance, etc. In practical applications, this solution can intelligently identify hot messages and optimize the access path. Through multi-level caching and dynamic partition adjustment, it significantly improves the throughput and response speed of message processing. It is particularly suitable for scenarios with significant hot access characteristics, such as social media hot events, e-commerce flash sales, etc. Through precise hot spot identification and efficient cache management, it realizes the performance optimization of the message processing system. The systematicness and innovation of this solution enable it to adapt to different scales of message processing requirements, and through continuous optimization and improvement, it realizes the comprehensive improvement of the message system.

[0071] As can be seen from the above description, the kafka storage-computation separation data processing method provided by the embodiment of this application can build a multi-layer data processing architecture, including four layers: network communication, task distribution, processing scheduling, and storage. Through the hierarchical design of the write-ahead log area and the main storage area, hot and cold data separation is achieved, and the intelligent scheduling of system resources is realized by combining the replica manager, coordinator, and load predictor. An adaptive compression algorithm and a verification mechanism are used to ensure data transmission efficiency and integrity. A message buffer is created to store hot data, and flexible access to hot and cold data is realized based on message positions. This method effectively solves the deficiencies of traditional technologies in aspects such as storage-computation coupling, storage efficiency, and load balance, and significantly improves the performance and scalability of the message queue system.

[0072] In an embodiment of the kafka storage-computation separation data processing method of this application, it may specifically include the following content:

[0073] Step S201: Divide the data processing architecture into call chains. Set up a socket listener in the network communication layer to handle client connection requests, create a message transmission channel and configure transmission parameters. Build a routing table in the task distribution layer to map the corresponding relationship between processors and message types, establish a task queue, set up a task priority scheduling mechanism, and distribute tasks to the corresponding processors according to the message type;

[0074] Step S202: Deploy a replica manager in the processing and scheduling layer to implement partition master-slave replication, configure a coordinator to manage partition allocation and consumer groups, deploy a load predictor to collect system resource usage data, create a stream repository in the storage layer, and divide the write-ahead log area and the main storage area. The write-ahead log area uses a solid-state drive storage medium, and the main storage area uses an object storage service as the persistent storage.

[0075] Optionally, this embodiment adopts a hierarchical call link design to achieve full-process tracking of requests from access to processing. In the network communication layer, an event-driven framework is built using the Reactor pattern. The main Reactor thread is responsible for receiving client connection requests, and the sub-Reactor thread pool processes data transmission and reception for established connections. High-performance IO multiplexing is achieved through the epoll mechanism, significantly improving the connection processing ability. Each connection is assigned a unique session ID to track the lifecycle of the request.

[0076] This embodiment implements an adaptive traffic control mechanism in the message transmission channel. The transmission rate is dynamically adjusted through the sliding window algorithm, and the window size is automatically adjusted according to network latency and packet loss rate. Transmission parameters include TCP send buffer size, receive buffer size, heartbeat interval, etc., which can be dynamically optimized according to the actual network conditions. At the same time, message fragmentation transmission is implemented, and large messages are sliced to ensure transmission reliability.

[0077] This embodiment designs an efficient task routing mechanism. The routing table adopts a multi-level hash structure. The first level is hashed based on the message topic, the second level is hashed based on the message type, and finally mapped to a specific processor. To improve the search efficiency, the routing table supports periodic optimization, and migrates the frequently accessed routing entries to the fast access area. The processor pool adopts an elastic scaling design and can dynamically adjust the number of processors according to the load situation.

[0078] This embodiment constructs an intelligent task scheduling system. The task queue adopts a multi-level feedback queue design, and tasks with different priorities enter different queue levels. Priority calculation considers multiple factors: the timeliness requirements of the message, the user service level, the queue waiting time, etc. Through dynamic time slice allocation, it is ensured that high-priority tasks can be processed quickly, while preventing low-priority tasks from starving.

[0079] This embodiment implements a reliable replica management mechanism. The replica manager implements master-slave replication based on an improved Raft protocol, introduces parallel replication optimization, and allows slave nodes to write multiple log entries in parallel. The replication process adopts a pipelined design, and the next batch of data is sent while waiting for the confirmation of the previous batch. At the same time, an intelligent failover mechanism is implemented, and when the master node fails, a new master node can be quickly elected.

[0080] This embodiment designs an innovative partition management strategy. The coordinator maintains a global partition allocation table and uses the consistent hashing algorithm for partition allocation. Multiple constraints are considered during the allocation process: broker load balancing, rack awareness, data locality, etc. When partition rebalancing occurs, the data transfer overhead is minimized through incremental migration. The consumer group management adopts a distributed session mechanism to ensure quick detection and handling in case of consumer failures.

[0081] This embodiment constructs an accurate load monitoring system. The load predictor collects system metrics through various means such as JMX and cgroups, including CPU usage, memory occupancy, disk I / O, network throughput, etc. The collected data is stored in a time series database, and multi-dimensional metric aggregation analysis is established. The prediction model uses an LSTM neural network, with input features including historical load data and system event logs, and the output predicts the resource usage trend within a future time window.

[0082] This embodiment implements an efficient storage management mechanism. The write-ahead log area utilizes the high IOPS characteristics of solid-state drives and adopts a multi-queue design to improve parallel write performance. The log files are sliced into fixed sizes, and memory mapping technology is used to accelerate read and write operations. The main storage area uses an object storage service to achieve heat tiered storage of data. Hot data is preferentially stored in the storage layer with higher performance, and cold data is automatically migrated to the storage layer with lower costs.

[0083] Through the above technological innovations, this embodiment effectively solves the key problems in the distributed message system: limited connection processing ability, unbalanced task scheduling, insufficient data reliability, etc. In practical applications, this solution can support large-scale message processing requirements. Through a multi-layer architecture and intelligent scheduling, the throughput and reliability of the system are significantly improved. It is particularly suitable for scenarios such as financial transactions and Internet of Things data processing. Through flexible storage strategies and reliable replication mechanisms, the overall performance of the message system is improved. The systematicness and innovativeness of this solution enable it to adapt to the application requirements of enterprises of different scales, and through continuous optimization and improvement, a comprehensive upgrade of the message processing system is achieved.

[0084] In an embodiment of the kafka storage-computation separation data processing method of this application, the following content may also be specifically included:

[0085] Step S301: Calculate the number of log slices based on the total capacity of the write-ahead log area, assign a unique identifier to each log slice, organize the log slices into a linked list structure in chronological order, set the message offset and timestamp index in the log slices, configure the reserved space threshold of the log slices, establish the mapping relationship between the log slices and physical storage devices, divide the main storage area into storage buckets according to the time range, and create metadata indexes for each storage bucket;

[0086] Step S302: Configure the master-slave replication parameters of the replica manager, establish a synchronization mechanism for partition replicas, build a partition allocation table and a consumer group management table in the coordinator, record consumer group member information and consumption progress. The load predictor collects system resource metrics such as processor utilization rate, memory occupancy rate, disk read / write rate, etc., constructs a time series prediction model to calculate the resource usage trend, and generates a load balancing strategy based on the prediction results.

[0087] Optionally, in this embodiment, a dynamic allocation strategy is adopted in log shard management. Calculate the initial number of shards by the ratio of the total capacity of the write-ahead log area to the base size of a single shard. The shard size adopts a logarithmic growth pattern, and the size of the new shard is 1.5 times that of the previous shard. This design can adapt to the growth trend of message traffic. Each shard is assigned a 64-bit unique identifier, which contains timestamp, sequence number, and node ID information to ensure uniqueness in a distributed environment.

[0088] This embodiment implements an efficient shard index structure. Log shards are connected by a doubly linked list, and each shard node contains reference pointers to the previous and next shards. A sparse index is established inside the shard, and an index entry is created every 4KB of message data. The index entry contains message offset, physical location, and timestamp information. The skiplist structure is used to organize the index entries, achieving fast lookup with an O(log n) time complexity. At the same time, an index cache for hot shards is maintained in memory to improve the lookup performance for frequent access.

[0089] This embodiment designs an innovative space management mechanism. The reserved space threshold is calculated dynamically, considering the historical write rate and the remaining capacity. When the shard utilization rate reaches the threshold, an asynchronous migration mechanism is triggered to migrate the old data to the main storage area. The migration process adopts a batch replication strategy, and the migration efficiency is improved by parallel transmission of multiple threads. At the same time, intelligent fragmentation is achieved, and small shards are merged regularly to optimize the storage space utilization rate.

[0090] This embodiment constructs a reliable storage bucket management. The main storage area is divided into storage buckets according to the daily time range, and each storage bucket contains multiple data files. The metadata index of the bucket adopts a B+ tree structure, and the index entry contains information such as time range, message offset range, file path, etc. The index storage space is optimized by prefix compression technology, and at the same time, an incremental update mechanism for the index is implemented to reduce the index maintenance overhead.

[0091] This embodiment implements an accurate replica synchronization mechanism. The replica manager adopts an improved Raft protocol and introduces parallel replication optimization. Replication parameters include batch size, number of replication threads, heartbeat interval, etc., and these parameters can be dynamically adjusted according to the network conditions. The synchronization process uses a pipeline design, allowing multiple batches of data to be sent simultaneously, significantly improving the replication efficiency. At the same time, an intelligent catch-up mechanism is implemented, and the lagging replicas can quickly synchronize the latest data.

[0092] This embodiment designs an innovative partition management strategy. The partition allocation table is implemented using a consistent hashing ring, and the allocation balance is improved through virtual node technology. The consumer group management table records the session information, partition allocation status, and consumption positions of consumers. The position management adopts a two-phase commit protocol to ensure that it can be restored to a consistent state in case of a failure. At the same time, an incremental adjustment of consumer rebalancing is implemented to minimize the impact of the rebalancing process on the service.

[0093] This embodiment constructs an intelligent load prediction system. Through multi-dimensional metric collection, a complete resource profile is established. The collected metrics include CPU usage rate, memory occupancy, disk I / O, etc. at the system level, and request queue length, processing latency, etc. at the application level. The prediction model uses an LSTM neural network, and the attention mechanism is used to capture the correlation between different metrics. The model training adopts a sliding window strategy, and the window size is dynamically adjusted according to the load change period.

[0094] This embodiment implements an adaptive load balancing strategy. The load score is calculated based on the prediction results, and the score calculation formula is:

[0095] Score = w1 CPU + w2 Memory + w3 IO + w4 Network,

[0096] where wi is the weight factor, and Network represents the network bandwidth rate. When the load score of a certain node exceeds the threshold, the load balancing operation is triggered. The balancing process takes into account the data locality principle and preferentially migrates the load to the node with data replicas to reduce the data transmission overhead.

[0097] Through the above technological innovations, this embodiment effectively solves the key problems in the distributed storage system: low storage space management efficiency, poor data synchronization performance, load imbalance, etc. In practical applications, this solution can support large-scale message storage and processing requirements. Through intelligent sharding management and predictive load balancing, the performance and reliability of the system are significantly improved. It is particularly suitable for scenarios that require high throughput and low latency, such as real-time data analysis, log collection, etc. Through multi-level optimization strategies, the overall performance of the storage system is improved. The systematicness and innovativeness of this solution enable it to adapt to the application requirements of enterprises of different scales. Through continuous optimization and improvement, a comprehensive upgrade of the storage system is achieved.

[0098] In an embodiment of the data processing method for separating storage and computing of kafka in this application, the following content may also be specifically included:

[0099] Step S401: Parse the message header and message body of the batch message data sent by the producer, count the data distribution characteristics of the message body, calculate data characteristic indicators such as the entropy value, repetition degree, and numerical range of the message body, establish a compression algorithm evaluation matrix, calculate the compression ratio and processing delay of different compression algorithms according to the data characteristic indicators, and select the optimal compression algorithm in the evaluation matrix;

[0100] Step S402: Apply the selected compression algorithm to the batch message data to generate a compressed byte stream, calculate the CRC32 checksum of the compressed byte stream, package the compressed data length, compression algorithm identifier, and checksum into a metadata header, assemble the metadata header and the compressed byte stream into a data block, and select an available log shard in the write-ahead log area to write the data block.

[0101] Optionally, this embodiment adopts the zero-copy technology in the message parsing stage and directly accesses the message data through memory mapping. The message header parsing adopts a fixed-length field design, including information such as message length, version number, topic, partition, etc., and quickly locates the positions of each field through displacement operations. The message body adopts variable-length encoding and is stored in the TLV (Type-Length-Value) format, supporting nested structures and adapting to complex message formats. The parsing process realizes parallel processing, and multiple parsing threads process different message batches simultaneously.

[0102] This embodiment realizes accurate data characteristic analysis. The entropy value calculation adopts a sliding window method, and the window size can be dynamically adjusted according to the message size.

[0103] The entropy value calculation formula is H = -∑(Pi×log2(Pi)), where Pi represents the probability of the byte value i appearing. The repeatability analysis uses an improved LZ77 algorithm. By maintaining a sliding window of a fixed size, repeated data segments are identified. The numerical range analysis uses histogram statistics, divides the numerical distribution into multiple intervals, and calculates the data density of each interval.

[0104] This embodiment designs an innovative compression algorithm evaluation mechanism. The evaluation matrix includes multiple dimensions: compression ratio, compression speed, decompression speed, memory occupancy, etc. Each dimension is assigned a weight coefficient, and the weight value can be dynamically adjusted according to the application scenario. For example, in scenarios with high real-time requirements, the weight of the compression speed will be increased accordingly. The evaluation process uses a sample testing method, and representative samples are extracted from the message batch for compression testing.

[0105] This embodiment constructs an adaptive algorithm selection strategy. According to the message characteristics and evaluation results, a decision tree model is established. The branch nodes of the decision tree include conditions such as entropy value thresholds and repeatability thresholds, and the leaf nodes correspond to specific compression algorithms. By traversing the decision tree, the most suitable compression algorithm can be quickly selected. At the same time, an algorithm caching mechanism is implemented, and message batches with similar characteristics can reuse the previous algorithm selection results.

[0106] This embodiment implements an efficient compression processing flow. After the algorithm is selected, a multi-threaded parallel compression strategy is adopted. The message batch is sliced according to a fixed size, and each thread is responsible for the compression task of one slice. The compression results are passed through a shared memory queue, avoiding data copying between threads. At the same time, dynamic adjustment of the compression level is implemented, and the appropriate compression level is selected according to the CPU load.

[0107] This embodiment designs a reliable data verification mechanism. The CRC32 verification is implemented using hardware acceleration. Utilizing the CRC32 instructions of modern CPUs, the calculation speed is significantly improved. The verification process covers the compressed data and the metadata header to ensure the integrity of the data. At the same time, an incremental verification mechanism is implemented. When a data block is partially modified, only the verification value of the modified part needs to be recalculated.

[0108] This embodiment constructs a complete metadata management. The metadata header uses a compact binary format, and the storage overhead is reduced through bit field technology. The compression algorithm identifier is represented by an enumeration value, supporting fast algorithm identification. The data length field uses variable-length coding to adapt to data blocks of different sizes. At the same time, version control of the metadata is implemented, supporting backward compatibility of the metadata format.

[0109] This embodiment implements an optimized data writing strategy. The selection of log shards is based on multiple factors: the remaining space of the shard, the writing load, data locality, etc. The writing process is asynchronous. After accumulating multiple data blocks through a write buffer, they are written in batches to improve the writing efficiency. At the same time, a write-ahead mechanism is implemented. Before the data is actually written, the metadata of the data block is first written to the write-ahead log to ensure data persistence.

[0110] Through the above technological innovations, this embodiment effectively solves the key problems in message compression processing: inaccurate selection of compression algorithms, low compression efficiency, insufficient guarantee of data integrity, etc. In practical applications, this solution can intelligently select the optimal compression algorithm. Through parallel processing and optimized writing strategies, it significantly improves the throughput and reliability of message processing. It is particularly suitable for large-scale message processing scenarios, such as log collection, monitoring data processing, etc. Through multi-level optimization strategies, it realizes the overall performance improvement of the message compression and storage system. The systematicness and innovativeness of this solution enable it to adapt to different types of message data. Through continuous optimization and improvement, it realizes the comprehensive improvement of message processing efficiency.

[0111] In an embodiment of the data processing method for separating storage and computing in Kafka of this application, it may specifically include the following content:

[0112] Step S501: Allocate solid-state drive storage space for the log shard, configure the storage parameters of the log shard, set the storage buffer size and the disk flushing policy, select a slave node based on the configuration information of the replica manager, establish a replication channel between the master and slave nodes, send the log shard data to the slave node in batches, and after the slave node verifies the data integrity, it returns a confirmation message to the master node;

[0113] Step S502: Establish a message data upload task queue, scan the confirmed replication data area in the log shard at a preset time interval, group and package the messages in the data area according to the time range, calculate the storage path of the data packet, asynchronously upload the data packet to the storage bucket corresponding to the main storage area through the object storage interface, and update the storage location index of the data packet.

[0114] Optionally, this embodiment adopts a dynamic management strategy for solid-state drive space allocation. It allocates continuous physical blocks through the direct I / O interface of the file system to avoid the additional overhead brought by the file system cache. The storage space is pre-allocated in a fixed size, with the initial size of each shard being 1GB, and dynamic expansion is supported. The allocation process uses an asynchronous mode, and multiple shards can be created in parallel to improve the resource initialization efficiency.

[0115] This embodiment realizes optimized storage parameter configuration. The storage buffer uses direct memory allocation, bypassing the JVM heap memory management. Each shard is equipped with an independent write buffer and a read-ahead buffer. The buffer size is dynamically adjusted according to the historical IO pattern. The write buffer is default set to 64MB, and the size of the read-ahead buffer is optimized based on the sequential read pattern. The disk flushing policy combines time threshold and data volume threshold, triggering the disk flushing operation when the accumulated data exceeds the threshold or reaches the time interval.

[0116] This embodiment designs a reliable replication channel mechanism. The replication channel is implemented based on the Netty framework and adopts the zero-copy transmission technology. The channel establishment process includes identity authentication and capability negotiation, supporting SSL encryption and compressed transmission. The data transmission adopts a pipeline design, allowing multiple batches to be sent simultaneously, and the size of each batch is dynamically adjusted according to the network conditions. The sending rate is controlled by a sliding window to avoid network congestion.

[0117] This embodiment constructs a complete data verification process. After receiving the data, the slave node first verifies the integrity of the data packet, including checksum verification and sequence number check. After passing the verification, the data is written to the local storage, and the dual-write mechanism is adopted to ensure data reliability. The confirmation message contains the batch ID and the write location information, and is returned to the master node asynchronously to reduce the confirmation delay.

[0118] This embodiment realizes efficient upload task management. The task queue adopts a priority queue structure, and the priority is calculated based on the time sensitivity of the data and the storage pressure. The queue processing adopts a multi-threaded model, and the number of processing threads is dynamically adjusted according to the number of CPU cores and the IO load. The failure retry mechanism is implemented during the task execution process, and the retry interval adopts the exponential back-off strategy.

[0119] This embodiment designs an innovative data packaging strategy. The grouping process considers multiple dimensions: time range, data volume, access mode, etc. The optimal packaging size is determined through aggregation analysis to balance storage efficiency and retrieval performance. The packaging process supports parallel processing, and the data in multiple time ranges can be packaged simultaneously. At the same time, an incremental packaging mechanism is implemented, only processing the newly added data areas.

[0120] This embodiment constructs an intelligent storage path calculation scheme. The path generation considers the time attribute and business attribute of the data, and adopts a multi-level directory structure. The path calculation adopts the consistent hashing algorithm to ensure that related data is stored in adjacent locations, improving the batch reading efficiency. At the same time, the path conflict handling is implemented, and the same-name conflict is resolved by appending a unique identifier.

[0121] This embodiment implements a reliable asynchronous upload mechanism. The object storage interface encapsulates retry and concurrency control logic, supports segmented upload and resume from breakpoint. The upload process adopts stream processing and improves the transmission efficiency through pipelined design. At the same time, upload progress tracking is implemented, supporting fine-grained progress query and cancellation operations. The selection of storage buckets is based on the data life cycle policy, supporting automatic migration of data between different storage levels.

[0122] Through the above technological innovations, this embodiment effectively solves the key problems in the distributed storage system: storage performance bottleneck, data replication latency, low upload efficiency, etc. In practical applications, this solution can support large-scale data storage and fast replication requirements, and significantly improves the throughput and reliability of the system through multi-level optimization strategies. It is particularly suitable for scenarios that require high-performance storage and disaster recovery, such as financial transactions, log archiving, etc. Through optimized storage strategies and reliable replication mechanisms, the overall performance of the storage system is improved. The systematicness and innovativeness of this solution enable it to adapt to the application requirements of enterprises of different scales, and achieve a comprehensive upgrade of the storage system through continuous optimization and improvement.

[0123] In an embodiment of the data processing method for separating storage and computing of Kafka in this application, the following specific content may also be included:

[0124] Step S601: Allocate memory space for the message buffer, construct a cache index table to record the mapping relationship between message positions and storage addresses, set the capacity limit and eviction policy of the buffer, establish a hot message scoring mechanism, calculate the message heat score based on the access frequency and time decay factor of the message, and load the message data with a heat score exceeding the preset threshold into the buffer;

[0125] Step S602: Monitor the read and write load data of the message partition. The load predictor trains a prediction model based on historical load data, calculates the load trend within the future time window, triggers partition expansion when the predicted load exceeds the preset threshold, redistributes the message data in the existing partition to the newly added partition according to the consistent hashing algorithm, and updates the partition routing table.

[0126] Optionally, this embodiment adopts a multi-level cache architecture for memory space allocation. Manage the message buffer through an off-heap memory pool to avoid GC pressure. The memory pool adopts a segmented lock design, divides the overall memory space into multiple regions, and each region is controlled by an independent lock to reduce thread contention. The allocation granularity of the buffer is dynamically adjusted according to the message size, and the buddy algorithm is used for memory block management to effectively reduce memory fragmentation.

[0127] This embodiment realizes an efficient cache index structure. A three-level index design is adopted: the first level is the mapping from sites to partitions, and a skip list is used to achieve fast lookup; the second level is the message index within the partition, adopting a B+ tree structure; the third level is the mapping of the memory address of the specific message, using a direct addressing table. This multi-level index structure can achieve an O(logn) lookup efficiency in a large number of messages and support range query operations at the same time.

[0128] This embodiment designs an innovative message heat score mechanism. The heat score calculation formula is:

[0129] Score = F×e^(-λt)×W, where F is the access frequency, λ is the time decay coefficient, t is the time interval since the last access, and W is the message weight. The weight factor takes into account the business priority and data size of the message. The scoring process uses a sliding window statistic, and the window size can be dynamically adjusted according to the business characteristics to accurately reflect the real-time heat change of the message.

[0130] This embodiment constructs an intelligent cache eviction strategy. Combining the LRU-K and ARC algorithms to implement an adaptive eviction mechanism. Two cache queues are maintained: a frequently accessed queue and a generally accessed queue. Messages enter the general queue when they are first accessed and are upgraded to the frequent queue when the number of accesses exceeds the threshold. The size ratio of the two queues is dynamically adjusted according to the access pattern, achieving the optimal utilization of the cache space.

[0131] This embodiment realizes an accurate load monitoring system. Collect multi-dimensional load metrics: message throughput, latency distribution, queue depth, etc. Data collection adopts an adaptive sampling strategy, increasing the sampling frequency when the load changes significantly. Storing historical load data through a time series database, establishing a multi-dimensional load portrait, and providing training data for the prediction model.

[0132] This embodiment designs an advanced load prediction model. Adopting an LSTM neural network structure, the input features include historical load metrics, time features (hours, weeks, holidays), and system events. The model captures the correlation between different features through a multi-head attention mechanism and accurately predicts short-term and medium-term load trends. The training process adopts an online learning method to continuously optimize the model parameters and adapt to the dynamic changes of the load pattern.

[0133] This embodiment constructs a reliable partition expansion mechanism. The expansion trigger conditions comprehensively consider multiple factors: predicted load, resource utilization, performance metrics, etc. The expansion process adopts a progressive strategy. First, create a new partition and preheat it, and then reallocate the message data through the consistent hashing algorithm. The hashing algorithm uses virtual node technology to ensure the balance of data distribution and minimize the amount of data migration.

[0134] This embodiment realizes an efficient data migration process. The migration process adopts a batch processing mode, sets up a migration task queue to manage migration requests, improves the migration efficiency through multi-threaded parallel processing, and each thread is responsible for migrating a batch of data. At the same time, a migration progress tracking and fault recovery mechanism is realized to ensure the reliability of the migration process.

[0135] Through the above technological innovations, this embodiment effectively solves the key problems in the distributed message system: low cache hit rate, inaccurate load prediction, poor expansion efficiency, etc. In practical applications, this solution can intelligently identify hot messages and optimize the cache strategy. Through accurate load prediction and efficient expansion mechanism, it significantly improves the response speed and scalability of the system. It is particularly suitable for scenarios with obvious access hotspots, such as social media push, real-time recommendation, etc. Through multi-level optimization strategies, the overall performance of the message system is improved. The systematicness and innovativeness of this solution enable it to adapt to the application requirements of enterprises of different scales, and through continuous optimization and improvement, the comprehensive upgrade of the message processing system is realized.

[0136] In an embodiment of the data processing method for separating storage and computing of kafka in this application, the following content may also be specifically included:

[0137] Step S701: The consumer sends a position query request to the coordinator. The coordinator returns the storage location information corresponding to the message position. The consumer searches for the target message data in the message buffer according to the storage location information. If the target message data does not exist in the buffer, it obtains the corresponding data packet from the main storage area, reads the metadata header in the data packet to parse the compression algorithm identifier, and calls the corresponding decompression method to restore the message data.

[0138] Step S702: Calculate the checksum of the decompressed message data and compare it with the checksum recorded in the metadata header. After the verification passes, return the message data to the consumer. After the consumer processes the message, it submits the consumption position to the coordinator. The coordinator records the consumption position in the position management table and periodically persists the position management table to the storage system.

[0139] Optionally, this embodiment adopts a multi-level cache mechanism for position query processing. The coordinator maintains an in-memory cache of position mapping, which is implemented using ConcurrentHashMap for fast lookup. The cache is organized by consumer group and partition, using a two-layer hash table structure. For the position information of hot queries, the access pressure on the coordinator is reduced through local caching. The consistency of the cache is maintained through a version number mechanism. When the position is updated, each consumer is notified to update the local cache through a broadcast mechanism.

[0140] This embodiment realizes intelligent storage location parsing. The storage location information adopts a compressed encoding format, including information such as storage level, file path, offset, etc. The parsing process uses a state machine design and can handle different versions of the encoding format. Through the locality principle of location information, it predicts and prefetches adjacent message locations that may be accessed, reducing subsequent query latency.

[0141] This embodiment designs an efficient cache lookup strategy. A multi-level index is used for lookup in the message cache area. First, a Bloom filter is used to quickly determine whether a message may exist, and then a skip list is used to locate the specific location. When the cache misses, asynchronous preloading is started, and subsequent messages that may be accessed are predicted and loaded according to the access pattern of the message, improving the cache hit rate.

[0142] This embodiment constructs a reliable data packet reading mechanism. When obtaining data packets from the main storage area, an asynchronous IO method is used, and the concurrent reading quantity is controlled through a thread pool. The reading of data packets supports range requests, and only the necessary data parts are obtained. For large data packets, a sharding reading strategy is implemented to avoid loading too much data into memory at one time.

[0143] This embodiment realizes an optimized decompression processing flow. The selection of the decompression algorithm is quickly located through the identifier in the metadata header, and multiple compression algorithms such as LZ4, Snappy, ZSTD, etc. are supported. The decompression process is managed by a thread pool. For a large number of small messages, a batch decompression strategy is adopted to improve efficiency. At the same time, a decompression cache is implemented, and the same compressed data block is only decompressed once.

[0144] This embodiment designs a strict data verification process. The CRC32 algorithm accelerated by hardware is used for checksum calculation, and the calculation speed is improved through SIMD instructions. The verification process covers the entire content of the message, including metadata and payload data. When the verification fails, a multi-level retry mechanism is implemented. First, it tries to reread the data. If it continues to fail, it obtains the data from the replica node.

[0145] This embodiment constructs a reliable site submission mechanism. Consumers submit sites asynchronously. After accumulating a certain number of sites in the local buffer, they are submitted in batches. The submission process realizes idempotency, and repeated submission of the same site will not cause errors. The site submission adopts a two-phase commit protocol to ensure consistency in a distributed environment.

[0146] This embodiment realizes an efficient site management strategy. The site management table adopts a segmented lock design to reduce the lock competition for concurrent updates. The table records detailed site information, including consumer group ID, topic, partition, site value, timestamp, etc. The persistence process adopts an incremental method, and only the changed site information is saved. The reliability of site data is ensured through the write-ahead log mechanism.

[0147] Through the above technological innovations, this embodiment effectively solves the key problems in distributed message consumption, such as high site query latency, low data decompression efficiency, and unreliable site management. In practical applications, this solution can support large-scale message consumption scenarios. Through multi-level caching and optimized data processing flows, it significantly improves the efficiency and reliability of message consumption. It is particularly suitable for scenarios that require high throughput and low latency, such as real-time data processing and event stream processing. Through multi-level optimization strategies, it realizes the overall performance improvement of the message consumption system. The systematicness and innovativeness of this solution enable it to adapt to the application requirements of enterprises of different scales. Through continuous optimization and improvement, it realizes the comprehensive upgrade of the message consumption system.

[0148] In order to effectively solve the deficiencies of traditional technologies in aspects such as memory-computation coupling, storage efficiency, and load balancing, and significantly improve the performance and scalability of the message queue system, this application provides an embodiment of a kafka memory-computation separation data processing device for all or part of the content of the data processing method for implementing the kafka memory-computation separation. See Figure 2 , the kafka memory-computation separation data processing device specifically includes the following content:

[0149] An architecture construction module 10, used to construct a multi-layer data processing architecture. The multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing scheduling layer, and a storage layer. The storage layer includes a stream storage repository. The stream storage repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate;

[0150] A message storage module 20, used to receive batch message data sent by a producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a check code, write the compressed message data into the log shards in the write-ahead log area. The log shards are stored using solid-state drives. Establish a master-slave replication relationship among multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shards to the main storage area;

[0151] A message cache module 30 is used to create a message cache area for storing hot message data, adjust the number of message partitions according to the prediction result of the load predictor, and a consumer reads the hot message data from the message cache area based on the message position. When the target message data does not exist in the message cache area, the consumer obtains cold data from the main storage area according to the message position and decompresses it. The coordinator records the consumption progress of the consumer and verifies the data integrity based on the checksum.

[0152] As can be seen from the above description, the data processing device with separated storage and computing provided by the embodiments of the present application can build a multi-layer data processing architecture, including four levels: network communication, task distribution, processing scheduling, and storage. The separation of hot and cold data is achieved through the hierarchical design of the write-ahead log area and the main storage area, and the intelligent scheduling of system resources is realized by combining the replica manager, the coordinator, and the load predictor. The adaptive compression algorithm and the verification mechanism are used to ensure the data transmission efficiency and integrity. A message cache area is created to store hot data, and flexible access to hot and cold data is realized based on the message position. This method effectively solves the deficiencies of traditional technologies in aspects such as coupled storage and computing, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system.

[0153] At the hardware level, in order to effectively solve the deficiencies of traditional technologies in aspects such as coupled storage and computing, storage efficiency, and load balancing, and significantly improve the performance and scalability of the message queue system, the embodiments of the present application provide an electronic device for implementing all or part of the content in the data processing method with separated storage and computing of kafka. The electronic device specifically includes the following:

[0154] A processor, a memory, a communication interface, and a bus; wherein, the processor, the memory, and the communication interface complete communication with each other through the bus; the communication interface is used to realize information transmission between the data processing device with separated storage and computing of kafka and related devices such as the core business system, the user terminal, and the relevant database. The logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., and this embodiment is not limited thereto. In this embodiment, the logic controller can be implemented with reference to the embodiments of the data processing method with separated storage and computing of kafka and the embodiments of the data processing device with separated storage and computing of kafka, and the content is incorporated herein, and the repeated parts will not be described again.

[0155] It can be understood that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0156] In practical applications, part of the data processing method of kafka storage and computing separation can be executed on the electronic device side as described above, or all operations can be completed in the client device. Specifically, it can be selected according to the processing capacity of the client device and the limitations of the user usage scenario, etc. This application does not make any limitations in this regard. If all operations are completed in the client device, the client device may further include a processor.

[0157] The above-mentioned client device may have a communication module (i.e., a communication unit), and can be communicatively connected to a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, it may also include a server of an intermediate platform, such as a server of a third-party server platform communicatively linked to the task scheduling center server. The server may include a single computer device, or may include a server cluster composed of multiple servers, or a server structure of a distributed device.

[0158] Figure 3 It is a schematic block diagram of the system composition of the electronic device 9600 according to an embodiment of the present application. As Figure 3 shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It should be noted that this Figure 3 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0159] In one embodiment, the function of the data processing method of kafka storage and computing separation may be integrated into the central processing unit 9100. Among them, the central processing unit 9100 may be configured to perform the following controls:

[0160] Step S101: Construct a multi-layer data processing architecture, the multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing scheduling layer and a storage layer, the storage layer includes a stream repository, the stream repository includes a write-ahead log area and a main storage area, multiple log shards are divided in the write-ahead log area, the main storage area realizes data persistence through an object storage service, the processing scheduling layer includes a replica manager, a coordinator and a load predictor, the replica manager is responsible for data replica synchronization, the coordinator is responsible for partition allocation and group management, and the load predictor establishes a prediction model based on the system resource utilization rate;

[0161] Step S102: Receive the batch message data sent by the producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a check code, write the compressed message data into the log shard in the write-ahead log area, where the log shard is stored using a solid-state drive, establish a master-slave replication relationship among multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shard to the main storage area;

[0162] Step S103: Create a message buffer area for storing hot message data, adjust the number of message partitions according to the prediction result of the load predictor, and the consumer reads the hot message data from the message buffer area based on the message position. When the target message data does not exist in the message buffer area, the consumer obtains cold data from the main storage area according to the message position and decompresses it. The coordinator records the consumption progress of the consumer and verifies the data integrity based on the check code.

[0163] As can be seen from the above description, the electronic device provided by the embodiments of the present application constructs a multi-layer data processing architecture, including four layers: network communication, task distribution, processing scheduling, and storage. The separation of hot and cold data is achieved through the hierarchical design of the write-ahead log area and the main storage area, and the intelligent scheduling of system resources is realized by combining the replica manager, the coordinator, and the load predictor. The adaptive compression algorithm and the check mechanism are used to ensure the data transmission efficiency and integrity. A message buffer area is created to store hot data, and flexible access to hot and cold data is realized based on the message position. This method effectively solves the deficiencies of traditional technologies in aspects such as memory-computation coupling, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system.

[0164] In another embodiment, the data processing device for separating memory and computing in Kafka can be separately configured from the central processing unit 9100. For example, the data processing device for separating memory and computing in Kafka can be configured as a chip connected to the central processing unit 9100, and the functions of the data processing method for separating memory and computing in Kafka are realized through the control of the central processing unit.

[0165] As Figure 3 shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It should be noted that the electronic device 9600 does not necessarily have to include all the components shown in Figure 3 ; in addition, the electronic device 9600 may further include components not shown in Figure 3 , and reference can be made to the prior art.

[0166] AsFigure 3 As shown, the central processing unit 9100, sometimes also referred to as a controller or operation control, may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives inputs and controls the operation of various components of the electronic device 9600.

[0167] Among them, the memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices. It can store the above information related to failures, and can also store programs for executing relevant information. And the central processing unit 9100 can execute the programs stored in the memory 9140 to achieve information storage or processing, etc.

[0168] The input unit 9120 provides inputs to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to supply power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display can be, for example, an LCD display, but is not limited thereto.

[0169] The memory 9140 can be a solid-state memory, for example, a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It can also be a memory that stores information even when powered off, can be selectively erased, and has more data. Examples of such a memory are sometimes referred to as EPROMs, etc. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 can include an application / function storage section 9142, which is used to store application programs and function programs or the processes for operating the electronic device 9600 through the central processing unit 9100.

[0170] The memory 9140 can also include a data storage section 9143, which is used to store data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage section 9144 of the memory 9140 can include various drivers of the electronic device for communication functions and / or for executing other functions of the electronic device (such as a messaging application, an address book application, etc.).

[0171] The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which can be the same as in the case of a conventional mobile communication terminal.

[0172] Based on different communication technologies, in the same electronic device, multiple communication modules 9110 can be provided, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module, etc. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby implementing normal telecommunication functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is also coupled to a central processor 9100, enabling recording on the local device through the microphone 9132 and playing back the sound stored on the local device through the speaker 9131.

[0173] An embodiment of the present application also provides a computer-readable storage medium capable of implementing all steps in the kafka storage-computation separation data processing method where the execution subject in the above embodiment is a server or a client. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements all steps of the kafka storage-computation separation data processing method where the execution subject in the above embodiment is a server or a client. For example, when the processor executes the computer program, it implements the following steps:

[0174] Step S101: Construct a multi-layer data processing architecture. The multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing scheduling layer, and a storage layer. The storage layer includes a stream repository, and the stream repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate;

[0175] Step S102: Receive the batch message data sent by the producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a check code, write the compressed message data into the log shards in the write-ahead log area. The log shards are stored using solid-state drives. Establish a master-slave replication relationship among multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shards to the main storage area;

[0176] Step S103: Create a message buffer for storing hot message data, adjust the number of message partitions according to the prediction result of the load predictor, and the consumer reads the hot message data from the message buffer based on the message position. When the target message data does not exist in the message buffer, the consumer obtains cold data from the main storage area according to the message position and decompresses it. The coordinator records the consumption progress of the consumer and verifies the data integrity based on the checksum.

[0177] As can be seen from the above description, the computer-readable storage medium provided by the embodiments of the present application constructs a multi-layer data processing architecture, including four layers: network communication, task distribution, processing scheduling, and storage. The separation of hot and cold data is achieved through the hierarchical design of the write-ahead log area and the main storage area. The intelligent scheduling of system resources is realized by combining the replica manager, the coordinator, and the load predictor. The adaptive compression algorithm and the verification mechanism are used to ensure the data transmission efficiency and integrity. A message buffer is created to store hot data, and flexible access to hot and cold data is realized based on the message position. This method effectively solves the deficiencies of traditional technologies in terms of memory-computation coupling, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system.

[0178] An embodiment of the present application also provides a computer program product capable of implementing all the steps in the data processing method for separating memory and computing of Kafka with the execution subject being a server or a client in the above embodiments. When the computer program / instructions are executed by a processor, the steps of the data processing method for separating memory and computing of Kafka are implemented. For example, the computer program / instructions implement the following steps:

[0179] Step S101: Construct a multi-layer data processing architecture, which includes a network communication layer, a task distribution layer, a processing scheduling layer, and a storage layer. The storage layer includes a stream storage repository, which includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate;

[0180] Step S102: Receive the batch message data sent by the producer, calculate the data characteristics of the batch message data, select a compression algorithm according to the data characteristics, perform compression encoding on the batch message data to generate a checksum, write the compressed message data into the log shard in the write-ahead log area. The log shard is stored on a solid-state drive. Establish a master-slave replication relationship among multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shard to the main storage area;

[0181] Step S103: Create a message buffer for storing hot message data. Adjust the number of message partitions according to the prediction result of the load predictor. The consumer reads the hot message data from the message buffer based on the message position. When the target message data does not exist in the message buffer, the consumer obtains cold data from the main storage area according to the message position and decompresses it. The coordinator records the consumption progress of the consumer and verifies the data integrity based on the check code.

[0182] As can be seen from the above description, the computer program product provided by the embodiments of the present application constructs a multi-layer data processing architecture, including four layers: network communication, task distribution, processing scheduling, and storage. The hierarchical design of the write-ahead log area and the main storage area realizes the separation of hot and cold data. The intelligent scheduling of system resources is achieved by combining the replica manager, the coordinator, and the load predictor. The adaptive compression algorithm and the verification mechanism are used to ensure the data transmission efficiency and integrity. A message buffer is created to store hot data, and flexible access to hot and cold data is realized based on the message position. This method effectively solves the deficiencies of traditional technologies in aspects such as memory-computation coupling, storage efficiency, and load balancing, and significantly improves the performance and scalability of the message queue system.

[0183] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, apparatus, or computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (apparatuses), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0185] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one or more of the processes Figure 1 one or more processes and / or blocks Figure 1 specified in the block or blocks.

[0186] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more of the processes Figure 1 one or more processes and / or blocks Figure 1 specified in the block or blocks.

[0187] In the present invention, specific embodiments are used to elaborate the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A data processing method for separating storage and computing in Kafka, characterized in that, The method includes: Constructing a multi-layer data processing architecture, which includes a network communication layer, a task distribution layer, a processing and scheduling layer, and a storage layer. The storage layer includes a stream repository, and the stream repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing and scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate; Receiving batch message data sent by a producer, calculating the data characteristics of the batch message data, selecting a compression algorithm according to the data characteristics, performing compression encoding on the batch message data to generate a check code, writing the compressed message data into the log shards in the write-ahead log area. The log shards are stored using solid-state drives. Establish a master-slave replication relationship among multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shards to the main storage area; Creating a message buffer area for storing hot message data, adjusting the number of message partitions according to the prediction result of the load predictor. Consumers read the hot message data from the message buffer area based on message positions. When the target message data does not exist in the message buffer area, the consumers obtain cold data from the main storage area according to the message positions and decompress it. The coordinator records the consumption progress of the consumers and verifies the data integrity based on the check code.

2. The data processing method for Kafka storage and computing separation according to claim 1, characterized in that, The constructing of the multi-layer data processing architecture, which includes a network communication layer, a task distribution layer, a processing and scheduling layer, and a storage layer. The storage layer includes a stream repository, and the stream repository includes a write-ahead log area and a main storage area, includes: Dividing the data processing architecture into call chains, setting up a socket listener in the network communication layer to handle client connection requests, creating a message transmission channel and configuring transmission parameters. Constructing a routing table in the task distribution layer to map the corresponding relationship between processors and message types, establishing a task queue, setting up a task priority scheduling mechanism, and distributing tasks to corresponding processors according to message types; Deploying a replica manager in the processing and scheduling layer to implement partition master-slave replication, configuring a coordinator to manage partition allocation and consumer groups, deploying a load predictor to collect system resource usage data. Creating a stream repository in the storage layer and dividing a write-ahead log area and a main storage area. The write-ahead log area uses a solid-state drive storage medium, and the main storage area uses an object storage service as persistent storage.

3. The data processing method of kafka memory and computing separation according to claim 1, characterized in that The dividing of multiple log shards in the write-ahead log area, the main storage area realizing data persistence through an object storage service, the processing and scheduling layer including a replica manager, a coordinator, and a load predictor, the replica manager being responsible for data replica synchronization, the coordinator being responsible for partition allocation and group management, and the load predictor establishing a prediction model based on the system resource utilization rate, includes: Calculate the number of log shards based on the total capacity of the write-ahead log area, assign a unique identifier to each log shard, organize the log shards into a linked list structure in chronological order, set message offsets and timestamp indexes in the log shards, configure the reserved space threshold of the log shards, and establish a mapping relationship between the log shards and the physical storage device. The main storage area is divided into storage buckets according to time ranges, and metadata indexes are created for each storage bucket; Configure the master-slave replication parameters of the replica manager, establish a synchronization mechanism for partition replicas, build a partition allocation table and a consumer group management table in the coordinator, and record consumer group member information and consumption progress. The load predictor collects system resource metrics such as processor utilization rate, memory occupancy rate, and disk read / write rate, constructs a time series prediction model to calculate resource usage trends, and generates a load balancing strategy based on the prediction results.

4. The data processing method of kafka memory and computing separation according to claim 1, wherein Receiving the batch message data sent by the producer, calculating the data characteristics of the batch message data, selecting a compression algorithm according to the data characteristics, performing compression encoding on the batch message data to generate a check code, and writing the compressed message data into the log shards in the write-ahead log area, including: Parse the message header and message body of the batch message data sent by the producer, count the data distribution characteristics of the message body, calculate data characteristic indicators such as the entropy value, repetition degree, and numerical range of the message body, establish a compression algorithm evaluation matrix, calculate the compression ratio and processing delay of different compression algorithms according to the data characteristic indicators, and select the optimal compression algorithm in the evaluation matrix; Apply the selected compression algorithm to the batch message data to generate a compressed byte stream, calculate the CRC32 check code of the compressed byte stream, package the compressed data length, compression algorithm identifier, and check code into a metadata header, assemble the metadata header and the compressed byte stream into a data block, and select an available log shard in the write-ahead log area to write the data block.

5. The data processing method of kafka storage and computing separation according to claim 1, characterized in that The log shards are stored using solid-state drives. Establish a master-slave replication relationship among multiple log shards according to the configuration of the replica manager, and asynchronously upload the message data in the log shards to the main storage area, including: Allocate solid-state drive storage space for the log shards, configure the storage parameters of the log shards, set the storage buffer size and disk flushing policy, select slave nodes based on the configuration information of the replica manager, establish a replication channel between the master and slave nodes, send the log shard data to the slave nodes in batches, and the slave nodes return confirmation information to the master node after verifying the data integrity; Establish a message data upload task queue, scan the confirmed replication data area in the log shards at a preset time interval, group and package the messages in the data area according to time ranges, calculate the storage path of the data packet, and asynchronously upload the data packet to the storage bucket corresponding to the main storage area through the object storage interface, and update the storage location index of the data packet.

6. The data processing method for Kafka storage and computing separation according to claim 1, wherein, Create a message buffer area, which is used to store hot message data, and adjust the number of message partitions according to the prediction results of the load predictor, including: Allocate memory space for the message buffer, build a cache index table to record the mapping relationship between message positions and storage addresses, set the capacity limit and eviction policy of the buffer, establish a hot message scoring mechanism, calculate the message heat score based on the access frequency and time decay factor of the message, and load the message data with a heat score exceeding the preset threshold into the buffer; Monitor the read and write load data of the message partition. The load predictor trains a prediction model based on historical load data, calculates the load trend within a future time window, triggers partition expansion when the predicted load exceeds the preset threshold, redistributes the message data in the existing partition to the newly added partition according to the consistent hashing algorithm, and updates the partition routing table.

7. The data processing method for Kafka storage and computing separation according to claim 1, wherein The consumer reads the hot message data from the message buffer based on the message position. When the target message data does not exist in the message buffer, the consumer obtains the cold data from the main storage area according to the message position and decompresses it. The coordinator records the consumption progress of the consumer and verifies the data integrity based on the checksum, including: The consumer sends a position query request to the coordinator, and the coordinator returns the storage location information corresponding to the message position. The consumer searches for the target message data in the message buffer according to the storage location information. If the target message data does not exist in the buffer, it obtains the corresponding data packet from the main storage area, reads the metadata header in the data packet to parse the compression algorithm identifier, and calls the corresponding decompression method to restore the message data; Calculate the checksum of the decompressed message data and compare it with the checksum recorded in the metadata header. After passing the verification, return the message data to the consumer. After the consumer processes the message, it submits the consumption position to the coordinator, and the coordinator records the consumption position in the position management table and periodically persists the position management table to the storage system.

8. A data processing device for Kafka storage and computing separation, characterized in that, The device includes: An architecture construction module for constructing a multi-layer data processing architecture. The multi-layer data processing architecture includes a network communication layer, a task distribution layer, a processing and scheduling layer, and a storage layer. The storage layer includes a stream storage repository, and the stream storage repository includes a write-ahead log area and a main storage area. Multiple log shards are divided in the write-ahead log area. The main storage area realizes data persistence through an object storage service. The processing and scheduling layer includes a replica manager, a coordinator, and a load predictor. The replica manager is responsible for data replica synchronization. The coordinator is responsible for partition allocation and group management. The load predictor establishes a prediction model based on the system resource utilization rate; A message storage module for receiving batch message data sent by a producer, calculating the data characteristics of the batch message data, selecting a compression algorithm according to the data characteristics, performing compression encoding on the batch message data to generate a checksum, writing the compressed message data into a log shard in the write-ahead log area. The log shard is stored using a solid-state drive, establishing a master-slave replication relationship between multiple log shards according to the configuration of the replica manager, and asynchronously uploading the message data in the log shard to the main storage area; The message cache module is used to create a message cache area, which is used to store hot message data, adjust the number of message partitions according to the prediction result of the load predictor, and consumers read the hot message data from the message cache area based on the message position. When the target message data does not exist in the message cache area, the consumers obtain cold data from the main storage area according to the message position and decompress it. The coordinator records the consumption progress of the consumers and verifies the data integrity based on the checksum.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the kafka storage and computing separation data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the kafka storage and computing separation data processing method according to any one of claims 1 to 7.

Images