A message controllable optimization method and system based on Kafka

Abstract

The present invention relates to the technical field of message queues, and in particular to a Kafka-based message controllable optimization method and system. The method comprises obtaining data attributes and calculating data priority values ​​according to the data attributes; constructing a position index cache mechanism based on priority data, persisting partition-offset data in a cache pool to an index table according to the position index cache mechanism, and prioritizing consumption of high-priority messages based on the persisted priority index. The method comprises setting a consumption control program architecture, isolating consumption queues, and introducing message deduplication operations; performing full-link performance tuning under the premise of prioritizing consumption of high-priority messages, and expanding business scenarios based on the tuned full-link. The present invention uses a consumption control program to pull messages in a priority-oriented manner, thereby breaking through the disorder of parallel consumption of Kafka partitions, enabling high-priority messages to be directly redirected to corresponding locations for priority processing, and significantly shortening consumption delays.

Description

Technical Field

[0001] The present invention relates to the technical field of message queues, and in particular to a Kafka-based message controllable optimization method and system. Background Art

[0002] As a high-throughput distributed publish-subscribe messaging system, Kafka achieves high concurrency and traffic reduction through its partitioning mechanism. It is widely used in scenarios such as log transmission, e-commerce data collection, and system monitoring. In its core architecture, producers send messages to different partitions of a topic, and consumers pull messages from the partitions through consumer groups. The partitioning mechanism allows data to be locally ordered within a single partition, but appear disordered overall. This feature can bring high performance advantages in scenarios where there is no global sorting requirement.

[0003] In the traditional Kafka architecture, each partition consumes in parallel and shares resources, and lacks a message priority distinction mechanism. When a partition contains important data, its acquisition time may be squeezed by low-priority data in other partitions, resulting in high-priority messages not being processed in time. In addition, Kafka's native design does not make semantic-level distinctions in the importance of messages, and all messages are processed equally at the consumer end. However, actual business often requires message priority division, and the traditional architecture needs to rely on the business layer to additionally implement sorting logic, increasing system complexity and latency. For example, if user payment messages in e-commerce systems cannot be processed before browsing records, transaction status updates may be delayed. Finally, Kafka's high throughput characteristics rely on partition parallel processing, and global sorting or priority mechanisms may introduce additional coordination overhead, resulting in performance degradation. Existing solutions are difficult to achieve a balance between message priority control and orderly consumption without significantly sacrificing throughput. At this stage, a message controllable optimization method and system based on Kafka is needed. Summary of the Invention

[0004] In order to solve the problems of performance and sorting contradiction at the traditional Kafka architecture level, lack of priority at the business level and disorder bottleneck at the performance level, the present invention provides a message controllable optimization method and system based on Kafka.

[0005] In the first aspect, the present invention provides a message controllable optimization method based on Kafka, which adopts the following technical solution:

[0006] A message controllable optimization method based on Kafka, comprising:

[0007] Get data attributes and calculate data priority values ​​based on the data attributes;

[0008] Building a location index cache mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain;

[0009] Persist the partition-offset data in the cache pool to the index table based on the position index cache mechanism, including defining the index table structure and setting the batch write strategy;

[0010] Prioritize consumption of high-priority messages based on persistent priority indexes, including setting up a consumption control program architecture, isolating consumption queues, and introducing message deduplication.

[0011] Perform full-link performance tuning, including tuning cache pool parameters, while prioritizing the consumption of high-priority messages.

[0012] Business scenarios are expanded based on the optimized full link, including using KafkaMirrorMaker to synchronize priority index tables across data centers.

[0013] Furthermore, the data priority value is calculated based on the data attributes, including extracting the data source, data classification, data type, data size, and data age of the message data as data attributes, calculating the priority value using weighted average, and finally performing level mapping on the priority value to obtain a numerical priority level. The priority value calculation formula is:

[0014] ,

[0015] in, Expressed as the weight coefficient of the i-th attribute, It is represented as the score of the i-th attribute.

[0016] Furthermore, the location index cache mechanism is constructed based on priority data, including adding a priority processing interceptor to the interceptor chain of the Kafka producer. The interceptor calls the priority judgment program to assign a priority level to the message. In the callback function when the message is successfully sent, the RecordMetadata object containing the partition and offset is obtained, and a priority level field is added to it to form extended metadata, and then a hierarchical cache pool is constructed to store the extended metadata.

[0017] Furthermore, the construction of a hierarchical cache pool to store extended metadata includes constructing an A-Cache module and a B-Cache module, wherein the A-Cache module pre-allocates fixed-size memory blocks, each block stores partition-offset data units of the same priority level, the capacity of a single partition-offset data unit is N records, and a lock-free queue is used to achieve fast writing. The B-Cache module dynamically allocates memory blocks to store large data units that exceed the A-Cache capacity, and uses an LRU algorithm to eliminate cold data. When the partition-offset data unit capacity in the A-Cache module is full, a batch write to the data priority index table is triggered, and the number of records written at a single time is N.

[0018] Furthermore, a new priority processing interceptor is added to the interceptor chain of the Kafka producer, including implementing the onSend method through the Kafka interceptor interface ProducerInterceptor, injecting the priority level label into the message header before message serialization, and parsing the metadata object in the callback method, and extracting the triple including the partition number, unique offset and timestamp.

[0019] Furthermore, the priority consumption of high-priority messages based on the persistent priority index includes building an independent consumption control program, polling the data priority index table from high to low priority, extracting the partition number, offset and priority level of unconsumed messages, and pulling messages through Kafka's seek (P, O) interface to achieve priority-driven consumption order control, allocate independent thread pools for different priority levels, and then adopt a priority preemption strategy. When the high-priority queue has a task, the low-priority thread suspends consumption, and adds a unique identifier in the message header to perform message deduplication.

[0020] Furthermore, full-link performance tuning is performed under the premise of prioritizing consumption of high-priority messages, including dynamic tuning of cache pool parameters, consumer-side performance optimization, and priority preemption and resource allocation. The cache pool parameters are dynamically adjusted, and the A-Cache module capacity is calculated based on the peak service message volume. The maximum capacity of the B-Cache module is set to M. When the cached data exceeds the set threshold, the LRU algorithm is triggered to forcibly eliminate the cold data unit. The A-Cache module capacity calculation formula is:

[0021] ,

[0022] Among them, N is the number of records stored in a single Unit, is the expansion factor.

[0023] Furthermore, the consumer-side performance optimization includes using a batch pull strategy to calculate the pull interval for high-priority consumer threads, and using adaptive pull for low-priority consumer threads to dynamically adjust the pull amount based on the partition Lag. The calculation formula for the dynamically adjusted pull amount is:

[0024] ,

[0025] in, It is expressed as a regulation factor. The current lag of a partition is expressed as the number of unconsumed messages in the partition, which is obtained in real time through the Kafka AdminClient.

[0026] Furthermore, the use of Kafka MirrorMaker to achieve cross-data center synchronization of the priority index table includes using the Kafka MirrorMaker component to build a priority index table synchronization link between the main data center and the disaster recovery data center, using a configuration file to define a priority level mapping function, converting the main center priority level into the disaster recovery center level, adopting an idempotent write mechanism, and filtering duplicate synchronization data based on the unique joint key of the index table to avoid redundant storage in a distributed environment.

[0027] The second aspect is a message controllable optimization system based on Kafka, including:

[0028] The data acquisition module is configured to: acquire data attributes and calculate data priority values ​​according to the data attributes;

[0029] The cache module is configured to: build a location index cache mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain;

[0030] The index conversion module is configured to persist the partition-offset data in the cache pool to the index table based on the position index cache mechanism, including defining the index table structure and setting the batch write strategy;

[0031] The conversion module is configured to prioritize the consumption of high-priority messages based on a persistent priority index, including setting up a consumption control program architecture, isolating consumption queues, and introducing message deduplication.

[0032] The optimization module is configured to optimize full-link performance, including tuning cache pool parameters, while prioritizing the consumption of high-priority messages.

[0033] The output module is configured to expand business scenarios based on the optimized full link, including using KafkaMirrorMaker to synchronize priority index tables across data centers.

[0034] In summary, the present invention has the following beneficial technical effects:

[0035] 1. This paper extracts five major attributes of data source, classification, type, size, and timeliness and establishes a weighted average model to convert abstract business priority requirements into computable quantitative indicators, solving the problem that Kafka's native architecture cannot identify the importance of messages.

[0036] 2. This invention integrates "priority marking" and "storage location tracking" by injecting priority tags into the Kafka producer interceptor and recording the message location. The layered cache pool (A-Cache pre-allocates fixed memory blocks, B-Cache dynamically manages large units) is combined with a batch write strategy to avoid the performance loss of writing a single piece of data. While ensuring Kafka's high throughput characteristics, it ensures real-time recording of priority data, and the index write delay can be stabilized at the microsecond level, thereby improving the processing efficiency and stability of the producer side.

[0037] 3. The present invention pulls messages in a priority-oriented manner through a consumption control program, breaking through the disorder of parallel consumption of Kafka partitions, so that high-priority messages can be directly jumped to the corresponding location for priority processing, significantly shortening consumption delays. The independent thread pool design and priority preemption strategy (high-priority threads obtain resources first) further ensure the timeliness of important data. At the same time, the message deduplication mechanism avoids repeated processing in a distributed environment, ensuring that business data is accurate.

[0038] 4. Cache pool parameters are dynamically adjusted based on business peaks, and the LRU algorithm is combined to eliminate cold data, avoiding memory waste and performance jitter. The consumer side adopts differentiated pull strategies for different priorities to reduce invalid network requests and improve overall throughput. Full-link optimization ensures priority control while minimizing the loss of Kafka's native performance, achieving a balance between functional expansion and performance.

[0039] 5. Use KafkaMirrorMaker to synchronize priority index tables across data centers, support custom priority level mapping, and meet the differentiated needs of different business scenarios under the distributed architecture. Synchronization link optimization (high-priority independent channels, idempotent writes) ensures cross-center data consistency and timeliness, and disaster recovery switching with short delays, ensuring business continuity and improving system reliability and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a schematic diagram of the overall process of a Kafka-based message controllable optimization method according to an embodiment of the present invention.

[0041] Figure 2This is a general technical architecture diagram of Kafka for a Kafka-based message controllable optimization method according to an embodiment of the present invention.

[0042] Figure 3 This is a logic diagram of an offset storage-high throughput cache algorithm in a Kafka-based message controllable optimization method according to an embodiment of the present invention.

[0043] Figure 4 This is a logical structure diagram of data priority index in a Kafka-based message controllable optimization method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below with reference to the accompanying drawings.

[0045] Example 1

[0046] Reference Figure 1 , a message controllable optimization method based on Kafka in this embodiment includes:

[0047] Get data attributes and calculate data priority values ​​based on the data attributes;

[0048] Building a location index cache mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain;

[0049] Persist the partition-offset data in the cache pool to the index table based on the position index cache mechanism, including defining the index table structure and setting the batch write strategy;

[0050] Prioritize consumption of high-priority messages based on persistent priority indexes, including setting up a consumption control program architecture, isolating consumption queues, and introducing message deduplication.

[0051] Perform full-link performance tuning, including tuning cache pool parameters, while prioritizing the consumption of high-priority messages.

[0052] Business scenarios are expanded based on the optimized full link, including using KafkaMirrorMaker to synchronize priority index tables across data centers.

[0053] Specifically, a message controllable optimization method based on Kafka includes the following steps:

[0054] S1. Obtain data attributes and calculate data priority values ​​based on the data attributes;

[0055] like Figure 1As shown, by extracting the five attributes of the message and building a weighted average model, the data priority is quantified and graded. On the Kafka producer side, the message metadata or payload is parsed through a custom interceptor (ProducerInterceptor) to extract the following five attributes: data source, data classification, data type, data size, and data timeliness. Each attribute is then converted into a standardized score on a 0-100 scale. Among them, the data source is specifically assigned 80-100 points to the core system (such as ERP), 0-20 points to the edge system, data classification is specifically assigned 90-100 points to the first-level permission data (such as transaction instructions), and less than 30 points to the public data, data type is specifically assigned 60-80 points to structured data (such as JSON), and 20-40 points to unstructured data, and the data size is specifically assigned using an inverse linear mapping: , valid when the maximum value is ≤5MB, if it exceeds, it is fixed to 20 minutes, and the data validity adopts direct proportional linear mapping , where it is valid when the actual time limit ≤ the benchmark time limit.

[0056] A weighted average calculation is performed based on the standardized scores of each attribute. First, a default weight matrix is ​​preset and the weighted average formula is applied to calculate the priority value:

[0057] ,

[0058] in, Expressed as the weight coefficient of the i-th attribute, It is expressed as the score of the i-th attribute, and then P is mapped to 5 priority levels (L):

[0059] ,

[0060] Among them, P represents the priority value. Through the interceptor's onSend method, the level tag (format: priority: L) is injected into the message header (MessageHeader) before the message is serialized, ensuring that the priority information is transmitted along the entire message link.

[0061] S2. Build a location index cache mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain;

[0062] like Figure 2 、 Figure 3As shown in the figure, in the process of Kafka producer sending messages, a priority processing interceptor is added by implementing the ProducerInterceptor interface. The interceptor intervenes in the processing before message serialization (onSend method) and after the message is successfully sent (onAcknowledgment callback): Message header label injection: In the onSend method, the message content is parsed and the priority judgment program of the business system (such as the weighted average algorithm) is called to assign a priority level L (0-4, with 0 being the highest) to the message. The level is written to the message header (MessageHeader) in the form of the label priority: L. This label is transmitted to the Kafka cluster with the message and becomes the core identifier for subsequent processing. Extended metadata generation: After the message is successfully sent to the partition, the RecordMetadata object returned by Kafka (including partition number partition, offset offset, and timestamp timestamp) is obtained through the onAcknowledgment callback, and the priority level field is added to it to form a custom PriorityRecordMetadata object, which binds the priority to the message storage location (partition + offset) to provide key data for subsequent index cache.

[0063] In the Kafka messaging system, producers need to record the partition and offset information of messages after sending them, so that consumers can pull them according to priority. Directly writing this metadata to the database one by one would lead to performance bottlenecks due to frequent I / O. Therefore, this embodiment designs a tiered cache pool called A-Cache and B-Cache. This improves data processing efficiency through in-memory caching and batch write mechanisms, while also preventing memory overflows.

[0064] The design and data storage mechanism of the hierarchical cache pool is based on the priority level obtained in step S1. When the A-Cache module starts, a set of fixed-size memory blocks are pre-allocated. Each block is used to store the partition-offset data unit (Unit) of the same priority. Each unit can accommodate up to N records. The calculation formula of N is Each Unit uses a lock-free queue (such as ConcurrentLinkedQueue) to implement multi-threaded concurrent writing to avoid performance loss caused by lock contention. When the number of records in the Unit reaches N, a batch write operation is triggered to store N records in the data priority index table at one time. This design reduces the database IO times to 1 / N of a single write, significantly improving persistence efficiency.

[0065] The B-Cache module dynamically allocates B-Cache memory blocks for units whose data volume exceeds the capacity of a single A-Cache block, such as metadata for large data messages. The size can be flexibly adjusted, such as 4KB-64KB. If the metadata unit size exceeds the capacity of a single A-Cache block, it is directly stored in the B-Cache. If there are no free blocks in the A-Cache, the new data will also be stored in the B-Cache. The B-Cache module uses the LRU (least recently used) algorithm to maintain memory blocks. When the memory usage exceeds a threshold (such as 80% of the total capacity), the cold data unit that has not been accessed for the longest time is automatically eliminated to free up space to accommodate new data. The specific working principle of the LRU algorithm is: metadata units that have not been accessed or processed for a long time are defined as cold data. If such data continues to occupy memory, it will reduce cache efficiency. The access order of all B-Cache data units is maintained, and each node contains a data unit reference and access timestamp. The most recently accessed unit is placed at the head of the linked list, and the unit that has not been accessed for the longest time is placed at the tail of the linked list. The key is the unique identifier of the data unit (such as a combination of priority + partition + offset), and the value is the reference to the linked list node. When a data unit is accessed (such as reading or writing), the linked list node is located through the hash table, removed from the current position, and inserted into the head of the linked list, marking it as the most recently accessed. When the B-Cache memory usage exceeds a preset threshold (such as 80% of the total capacity), the elimination mechanism is triggered, starting from the tail of the linked list, and the nodes that have not been accessed for the longest time are eliminated in sequence until the memory usage drops below the threshold.

[0066] The coordinated scheduling system with A-Cache automatically selects a cache module based on the unit data size. If the unit size is ≤ the A-Cache single block size and there are free blocks in A-Cache, the data is stored in A-Cache; otherwise, it is stored in B-Cache. This mechanism balances the efficiency of fixed blocks with the flexibility of dynamic blocks. The A-Cache module capacity calculation formula is:

[0067] ,

[0068] Among them, N is the number of records stored in a single Unit. is the expansion factor, and the capacity of the B-Cache module is twice.

[0069] The newly added PriorityRecordMetadata object is classified by priority L and first attempts to store it in the unit of the corresponding priority in the A-Cache. If the unit capacity is not full, it continues to append records. If it is full, it triggers a batch write to the index table and resets the unit. If the unit size exceeds the A-Cache limit or there are no free blocks in the A-Cache, it is stored in the B-Cache module, and the life cycle is managed by the LRU algorithm.

[0070] Finally, asynchronous backup management is performed on the eliminated cold data. When the B-Cache memory usage exceeds the threshold, the LRU algorithm eliminates the cold data unit from the tail of the linked list, triggering the asynchronous backup process. The eliminated data unit is written to the local file system. The backup file is stored in layers according to priority and time dimensions. The path format is: / cache_backup / priority_L / YYYYMMDD / HHMM / unit_XXX.dat. The file name contains the key information of the data unit. The backup file is stored in binary format and contains complete information about the data unit. Finally, asynchronous backup uses two blocking queues (backupQueue and completedQueue) to implement the production-consumption model: backupQueue stores data units to be backed up, and completedQueue stores data units that have been backed up.

[0071] After that, cold data backup and recovery and re-entry into B-Cache operations are performed. When consumers query data, if neither A-Cache nor B-Cache hits, the backup file is searched by priority and offset, the backup file is read and deserialized into a DataUnit object, and the data integrity is verified by CRC32 checksum. Before recovery, the B-Cache memory usage is checked. If it exceeds the threshold, a round of LRU elimination of old data is triggered to make room for the newly restored data. A unique identifier of the data unit is generated for the restored data. If the identifier already exists in B-Cache, the old node is first removed from the linked list, and the restored data unit is inserted into the LRU linked list head as the latest access node, and the lruMap hash table reference is updated to ensure priority hit the next time it is accessed. The restored data unit re-participates in LRU elimination. If it is eliminated again, the backup process is repeated. After the data is successfully restored to B-Cache, the corresponding backup record is deleted from the completedQueue queue.

[0072] S3. Persist the partition-offset data in the cache pool to the index table based on the location index cache mechanism, including defining the index table structure and setting the batch write strategy.

[0073] The index table contains the priority level (priority_level), partition number (partition_id), offset (offset), and timestamp fields. A joint unique index is established on (priority_level, partition_id, offset) to ensure that the position index of each message is unique and unique. Database batch insert technology (such as JDBC batch processing) is used to encapsulate the unit data in the buffer pool into SQL batch statements (for example, inserting 1024 records at a time), reducing the number of database interactions. The example process is: the buffer pool collects units at full capacity → generates batch SQL → calls executeBatch() to execute → clears the units and reuses them.

[0074] Among them, the priority level is calculated by the weighted average algorithm in step S1. The partition number is the Kafka partition number where the message is stored. The offset is the unique offset of the message in the partition. The location identifier provided by Kafka natively ensures the message sequence. The timestamp is the message production timestamp, which is used for time range query and data timeliness management. A joint unique index is established on the (priority_level, partition_id, offset) field to ensure that records with the same priority, partition and offset are stored only once, avoiding index redundancy caused by repeated message sending. A normal index is established on the timestamp field to speed up the query of high-priority messages by time range.

[0075] The cache pool triggers batch writes under the following conditions: Specifically, in the A-Cache module, pre-allocated fixed-size memory blocks (e.g., 8KB / block) are used to store partition-offset data units of the same priority level. When the number of records stored in a single unit reaches the preset capacity N (e.g., 1024), a batch write is triggered. For example, when a unit for level 0 messages is full of 1024 records, the data is immediately written to the index table in batches.

[0076] B-Cache module: Dynamically allocated memory blocks are used to store large data units that exceed the A-Cache capacity (such as messages that occupy a large amount of space per record). When the B-Cache memory usage exceeds a threshold (such as 80% of the total capacity) or the idle time of a data unit exceeds a preset threshold (such as 5 seconds), a forced write is triggered to free up memory space.

[0077] Next, we set up the batch write execution logic, organizing the Unit data in the cache pool into a batch insert dataset based on the index table field order. For example, we convert the fully populated Units in A-Cache into a list of 1024 records, each corresponding to a row in the index table. Using the database's batch insert interface (such as MySQL's LOADDATAINFILE or PostgreSQL's COPY command), we submit the dataset to the index table all at once. Compared to single row inserts, this operation reduces the number of database-application interactions from N to 1, significantly reducing network latency and I / O overhead. After a successful batch write, the corresponding Unit space is cleared to allow for new data to be repopulated. If the write fails (e.g., due to a database connection interruption), the Unit data is restored to the cache pool and retried upon the next trigger condition.

[0078] Then, a data consistency and disaster recovery mechanism was introduced, and database transactions were enabled during the batch write process:

[0079] After starting a transaction, perform a batch insert operation. If all records are successfully written, the transaction is committed. If it fails midway, the transaction is rolled back and the unsuccessfully written data is stored back in the cache pool to avoid index inconsistencies caused by partial data writing. For example, if 500 out of 1024 records fail to be written, the entire batch operation will be rolled back and all data will be put back into the cache pool and wait for retry.

[0080] Cache data that is not written to the index table in time (such as unfilled units in A-Cache and cold data in B-Cache) is asynchronously backed up to the local file system and stored in priority- and time-partitioned files ( / cache_backup / priority_0 / 20250606 / unit_001.dat). The backup file contains binary serialization of the data, ensuring rapid recovery in the event of a failure.

[0081] When the system restarts, it scans the local backup file and parses out records that haven't been written to the index table. It then uses Kafka's AdminClient to obtain the current maximum offset for each partition, filters out unconsumed records (i.e., records with an offset greater than the current partition offset), and rewrites them to the index table in batches. This process ensures that index data that wasn't persisted during the outage isn't lost and that already consumed messages aren't processed repeatedly. Finally, the capacity N of a single unit is dynamically adjusted based on the message characteristics of the business scenario:

[0082] ,

[0083] in, "A-Cache Single Block Size" represents the number of bytes in the fixed-size memory blocks preallocated by the A-Cache module. "Average Single Record Size" represents the average number of bytes required to store a partition-offset data record. Furthermore, the index table structure and batch write logic support mainstream relational databases, adapting to the characteristics of different databases through an abstract data access layer. For MySQL, the rewriteBatchedStatements=true parameter is used to optimize batch insert performance. For Oracle, the FORALL statement is used to execute batch operations in parallel. For PostgreSQL, the PREPARE and EXECUTE statements are used to precompile batch insert statements to reduce syntax parsing overhead.

[0084] S4. Prioritize the consumption of high-priority messages based on a persistent priority index, including setting up a consumption control program architecture, isolating consumption queues, and introducing message deduplication.

[0085] like Figure 4 As shown in the figure, priority index polling: The system periodically queries the data priority index table for records with an "unconsumed" status in descending priority order (0 → 1 → 2 → 3 → 4). For example, the index of level 0 and level 1 messages is queried every 100 milliseconds, and low-priority messages are queried every 500 milliseconds to ensure that high-priority data is processed first.

[0086] Directed consumption uses the seek(partition, offset) interface provided by Kafka to directly jump to the specified offset in the target partition and pull messages. Traditional Kafka consumption processes read sequentially in partition order, but this solution uses an index table to obtain the precise location of messages (partition number + offset), implementing a "priority-first" consumption order control. For example, if the index table contains level-0 messages at offset 1000 for partition 1 and offset 2000 for partition 2, the control program will prioritize pulling messages from these two locations, rather than consuming them starting from partition 0 in partition order. The consumption status is updated, and upon successful consumption of a message, the corresponding record in the index table is immediately marked as "consumed" to avoid duplicate processing.

[0087] Traditional Kafka consumption is processed sequentially by partition, with no priority difference between messages in different partitions. This solution prioritizes messages using an index table. Index records for high-priority messages are polled and pulled first, thus overcoming partition restrictions. For example, level-0 messages may be distributed across multiple partitions. The control program collects all unconsumed level-0 indexes and pulls them sequentially in offset order. This ensures that messages within the same priority level are processed in partition order, while messages of different priorities are processed based on their level.

[0088] Then, a hierarchical thread pool architecture and resource allocation are carried out. The high-priority thread pool uses a fixed-size thread pool (such as 5 threads). Each thread is bound to an independent task queue (capacity 100). The independent queue ensures that high-priority messages will not be blocked by low-priority tasks. For example, the task queue of level 0 messages only stores consumer tasks corresponding to the level 0 index. The thread pool focuses on processing tasks in this queue to avoid mixing with other priority levels. The fixed number of threads is equal to 1.5 times the number of CPU cores to improve parallel processing capabilities.

[0089] The low-priority thread pool uses a dynamic thread pool (such as a cacheable thread pool). Multiple priorities share a task queue, and the number of threads is automatically adjusted according to the amount of tasks. For example, when there are fewer low-priority tasks, the thread pool automatically recycles idle threads to save resources; when there is a surge in tasks, new threads are dynamically created to handle them.

[0090] Then, we set the priority preemption policy and implement resource preemption through the thread scheduling mechanism at the operating system level: when there are tasks to be processed in the task queue of the high-priority thread pool, the low-priority thread will actively release the CPU time slice and suspend consumption. The specific implementation method is:

[0091] When a high-priority thread is executing, the thread priority is set to the highest (such as Thread.MAX_PRIORITY in Java). The low-priority thread periodically checks the status of the high-priority queue during operation. If a task is found, it actively gives up the CPU through Thread.yield() until the high-priority queue is empty.

[0092] Finally, message deduplication is performed. During the message production phase, a producer interceptor adds a globally unique identifier (such as a UUID) to each message and stores it in the message header in the format message-id:UUID. For example, the unique identifier of a message is message-id:ab12cd34-5ef6-7gh8-9ij0-klm1nop2q3r4. The consumer maintains a local cache (such as a memory-based hash table) to record the UUIDs of processed messages. The cache capacity is set to 100,000 entries, and an LRU (least recently used) algorithm is used to eliminate old records. The cache expiration time is the maximum lifespan of the message (such as 7 days) to avoid long-term memory usage.

[0093] After receiving the message, the consumer first extracts the UUID from the message header and checks whether it exists in the local cache: if it exists, it means that the message has been processed and is discarded directly without performing any business logic processing; if it does not exist, the message is submitted to the thread pool for processing and the UUID is stored in the cache.

[0094] When a seek(partition, offset) operation fails (for example, because the partition has been deleted or the offset exceeds the maximum offset of the current partition), the consumer control program records an error log, skips the current index record, and continues processing the next valid record, preventing the entire consumer process from being blocked by a single exception.

[0095] Disable automatic offset submission through Kafka's auto.commit mechanism and switch to manual submission: commit the current consumption offset only after the message is successfully processed and the index table status is updated, ensuring that processed data is not lost during fault recovery.

[0096] S5. Perform full-link performance tuning, including tuning cache pool parameters, while prioritizing consumption of high-priority messages.

[0097] Maintain a bidirectional linked list to record the access order of data units in B-Cache. When the cache occupancy exceeds 80% of the maximum capacity, the cold data units that have not been accessed for the longest time are eliminated from the tail of the linked list until the occupancy rate drops below 70%. If the eliminated data units are not written to the index table, the number of messages pulled by the high-priority thread (level 0, level 1) at a time is set to By reducing the number of interactions between the Kafka client and the Broker, the throughput is improved. The calculation formula for the pull interval is:

[0098] ,

[0099] in, Indicates the number of messages pulled at a time and sets a timeout for the pull operation milliseconds, if not reached Messages are returned in advance, and low-priority threads use adaptive pull. Before pulling, the Lag of the target partition is obtained through AdminClient to calculate the pull amount. , the dynamic adjustment formula is:

[0100] ,

[0101] in, It is expressed as a regulation factor. The current Lag of a partition is expressed as the number of unconsumed messages in the partition. It is obtained in real time through Kafka AdminClient. The physical meaning of Lag is expressed as follows: Partition Lag is the number of unconsumed messages in a Kafka partition. The calculation formula is:

[0102] ,

[0103] For example, if the maximum offset of a partition is 10,000 and the consumer has consumed 8,000 messages, then Lag = 2,000 messages, reflecting the partition's backlog. Partition Lag-driven pull volume adjustment, dynamic calculation of cache pool capacity, and flexible allocation of CPU time slices enable the system to automatically balance performance and resource utilization during traffic fluctuations without manual intervention. Consumers pull messages by specifying a partition and offset. In this embodiment, high-priority threads directly locate target messages in the partition through the seek(partition, offset) interface, breaking through the disorder of traditional partitioned parallel consumption. Partition Lag is calculated based on the difference between each partition's maximum offset and the consumer's offset, reflecting the partition's message backlog and serving as a key basis for adaptive pull by low-priority threads. Lag is calculated separately for each partition.

[0104] Partitions are the basis for Kafka to implement distributed storage and parallel processing. They are set on the Broker nodes of the Kafka cluster. In this solution, partitions are not only the physical units of message storage, but also the key dimensions of the priority pull strategy. Through mechanisms such as partition lag calculation and partition-oriented pull, high-priority messages can be consumed preferentially across partitions. At the same time, adaptive pull is used to balance the load of each partition, ultimately improving the timeliness of message processing while ensuring throughput.

[0105] When the task queue backlog in the high-priority thread pool exceeds a threshold (for example, when the remaining queue capacity is less than 20%), the system uses the operating system's CPU scheduler to reduce the CPU time slices of low-priority threads and allocate the freed-up time slices to high-priority threads. When idle (for example, when the task queue is empty), low-priority threads preload some high-priority index data into the memory cache, reducing the cold start delay of high-priority tasks. A-Cache adopts a garbage collection-free design. It uses Direct Memory or object pools (such as the Apache Commons Pool) to pre-allocate memory blocks, avoiding pauses caused by Java heap memory allocation and garbage collection (GC). This embodiment uses Unsafe.allocateMemory() to directly allocate 8KB memory blocks, eliminating the impact of garbage collection on high-priority tasks. B-Cache uses generational recycling, and the A-Cache capacity dynamically adjusts with peak message volume to ensure that the producer-side index cache capacity matches the message production rate. B-Cache's LRU eviction mechanism prevents memory overflows and reserves space for high-priority metadata.

[0106] S6. Expand business scenarios based on the optimized full link, including using KafkaMirrorMaker to synchronize priority index tables across data centers.

[0107] Deploy the Kafka MirrorMaker component between the primary data center and the disaster recovery data center to form a unidirectional or bidirectional synchronization link:

[0108] Data outflow from the master center: When the master center's priority index table changes (such as adding a partition-offset record for a high-priority message), the MirrorMaker producer captures the changed data, serializes it into a Kafka message, and sends it to a dedicated topic.

[0109] Data inflow to the disaster recovery center: The MirrorMaker consumer in the disaster recovery center subscribes to the topic, parses the message content, and writes it into the local priority index table to ensure consistency of index data in both locations.

[0110] Synchronization granularity: Only synchronize "unconsumed" records in the index table to avoid invalid transmission of consumed data and reduce network traffic.

[0111] Different data centers may need to adjust their priority level definitions due to differing business rules. Mapping rules are defined in the configuration file to convert the primary center's priority level (L) to the disaster recovery center's level (L'). Data classified as "relatively important (Level 0)" in the primary center must be upgraded to "important (Level 1)" in the disaster recovery center to comply with the disaster recovery's risk control rules. Data classified as "general (Level 3)" in the primary center remains at its original level in the disaster recovery center.

[0112] The priority index table ensures idempotence of cross-center synchronization by combining a unique key (priority_level, partition_id, offset):

[0113] Before writing synchronized data, the disaster recovery center first checks the local index table for records with the same joint key. If so, the data has already been synchronized and the write operation is skipped. If not, the insert operation is performed. This mechanism avoids duplicate data caused by network retries, cluster switching, etc. For example, if the primary center sends the same index change repeatedly, the disaster recovery center only processes it once.

[0114] By adding database indexes (such as B-tree indexes) for joint unique keys, the response time of deduplication queries is controlled at the millisecond level to ensure that synchronization efficiency is not affected. For batch synchronized index data, a unique key set is first built in memory (such as using HashSet), and existing records are filtered out before being written to the database in batches to reduce disk IO times. A bidirectional MirrorMaker link is established between the primary center and the disaster recovery center. Under normal circumstances, data is synchronized one-way from the primary center to the disaster recovery center. When the primary center fails, manual or automatic switching is performed to synchronize data from the disaster recovery center to the primary center. Independent Kafka topic partitions are assigned to high-priority index records. Message compression is enabled for low-priority index data through Kafka's partition priority or external traffic control tools.

[0115] Example 2

[0116] This embodiment differs from Embodiment 1 in that it provides a Kafka-based message controllable optimization system, including:

[0117] The data acquisition module is configured to: acquire data attributes and calculate data priority values ​​according to the data attributes;

[0118] The cache module is configured to: build a location index cache mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain;

[0119] The index conversion module is configured to persist the partition-offset data in the cache pool to the index table based on the position index cache mechanism, including defining the index table structure and setting the batch write strategy;

[0120] The conversion module is configured to prioritize the consumption of high-priority messages based on a persistent priority index, including setting up a consumption control program architecture, isolating consumption queues, and introducing message deduplication.

[0121] The optimization module is configured to optimize full-link performance, including tuning cache pool parameters, while prioritizing the consumption of high-priority messages.

[0122] The output module is configured to expand business scenarios based on the optimized full link, including using KafkaMirrorMaker to synchronize priority index tables across data centers.

[0123] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A message controllable optimization method based on Kafka, characterized in that: include: Get data attributes and calculate data priority values ​​based on the data attributes; Building a location index cache mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain; The priority data-based location index cache mechanism is constructed, including adding a priority processing interceptor to the interceptor chain of the Kafka producer. The interceptor calls the priority judgment program to assign a priority level to the message. In the callback function of the successful message sending, a RecordMetadata object containing the partition and offset is obtained and a priority level field is added to it to form extended metadata. Then, a hierarchical cache pool is constructed to store the extended metadata. The construction of the hierarchical cache pool storage extended metadata includes constructing an A-Cache module and a B-Cache module, wherein the A-Cache module pre-allocates fixed-size memory blocks, each block stores partition-offset data units of the same priority level, a single partition-offset data unit has a capacity of N records, and uses a lock-free queue to achieve fast writing. The B-Cache module dynamically allocates memory blocks for storing large data units that exceed the A-Cache capacity, and uses a LRU algorithm to eliminate cold data. When the partition-offset data unit capacity in the A-Cache module is full, a batch write to the data priority index table is triggered, and the number of records written at a single time is N; Persist the partition-offset data in the cache pool to the index table based on the position index cache mechanism, including defining the index table structure and setting the batch write strategy; Prioritize consumption of high-priority messages based on persistent priority indexes, including setting up a consumption control program architecture, isolating consumption queues, and introducing message deduplication. Perform full-link performance tuning, including tuning cache pool parameters, while prioritizing the consumption of high-priority messages. The full-link performance tuning is performed under the premise of prioritizing the consumption of high-priority messages, including dynamic tuning of cache pool parameters, consumer-side performance optimization, priority preemption, and resource allocation. The cache pool parameters are dynamically adjusted, and the A-Cache module capacity is calculated based on the peak service message volume. The maximum capacity of the B-Cache module is set to M. When the cached data exceeds the set threshold, the LRU algorithm is triggered to forcibly eliminate cold data units. The A-Cache module capacity calculation formula is: ， Among them, N is the number of records stored in a single Unit, is the expansion coefficient; Business scenarios are expanded based on the optimized full link, including using KafkaMirrorMaker to synchronize priority index tables across data centers.

2. The message controllable optimization method based on Kafka according to claim 1 is characterized in that: The data priority value is calculated based on the data attributes, including extracting the data source, data classification, data type, data size, and data age of the message data as data attributes, calculating the priority value using weighted average, and finally performing level mapping on the priority value to obtain a numerical priority level. The priority value calculation formula is: ， in, Expressed as the weight coefficient of the i-th attribute, It is represented as the score of the i-th attribute.

3. The message controllable optimization method based on Kafka according to claim 1 is characterized in that: The method adds a new priority processing interceptor to the interceptor chain of the Kafka producer, including implementing the onSend method through the Kafka interceptor interface ProducerInterceptor, injecting the priority level label into the message header before message serialization, and parsing the metadata object in the callback method and extracting the triple including the partition number, unique offset and timestamp.

4. The message controllable optimization method based on Kafka according to claim 1 is characterized in that: The method of giving priority to consuming high-priority messages based on a persistent priority index includes building an independent consumption control program, polling a data priority index table from high to low priority, extracting the partition number, offset and priority level of unconsumed messages, pulling messages in a targeted manner through Kafka's seek (P, O) interface, realizing priority-driven consumption order control, allocating independent thread pools for different priority levels, and then adopting a priority preemption strategy. When a high-priority queue has a task, a low-priority thread suspends consumption, and adds a unique identifier in the message header to perform message deduplication.

5. The message controllable optimization method based on Kafka according to claim 1 is characterized in that: The consumer-side performance optimization includes using a batch pull strategy to calculate the pull interval for high-priority consumer threads, and using adaptive pull for low-priority consumer threads to dynamically adjust the pull amount based on the partition Lag. The calculation formula for the dynamically adjusted pull amount is: ， in, It is expressed as a regulation factor. The current lag of a partition is expressed as the number of unconsumed messages in the partition, which can be obtained in real time through KafkaAdminClient.

6. The message controllable optimization method based on Kafka according to claim 1 is characterized in that: The cross-data center synchronization of the priority index table is achieved by using Kafka MirrorMaker, including using the Kafka MirrorMaker component to build a priority index table synchronization link between the primary data center and the disaster recovery data center, using a configuration file to define a priority level mapping function, converting the primary center priority level into the disaster recovery center level, adopting an idempotent write mechanism, and filtering duplicate synchronization data based on the unique joint key of the index table to avoid redundant storage in a distributed environment.

7. A message controllable optimization system based on Kafka, executing the method according to claim 1, characterized in that: include: The data acquisition module is configured to: acquire data attributes and calculate data priority values ​​according to the data attributes; The cache module is configured to: build a location index cache mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain; The index conversion module is configured to persist the partition-offset data in the cache pool to the index table based on the position index cache mechanism, including defining the index table structure and setting the batch write strategy; The conversion module is configured to prioritize the consumption of high-priority messages based on a persistent priority index, including setting up a consumption control program architecture, isolating consumption queues, and introducing message deduplication. The optimization module is configured to optimize full-link performance, including tuning cache pool parameters, while prioritizing the consumption of high-priority messages. The output module is configured to expand business scenarios based on the optimized full link, including using KafkaMirrorMaker to synchronize priority index tables across data centers.

Images