Message controllable optimization method and system based on kafka

Abstract

The invention relates to the technical field of message queues, in particular to a message controllable optimization method and system based on kafka, and the method comprises the steps: obtaining a data attribute, and calculating a data priority value according to the data attribute; the method comprises the steps of establishing a position index cache mechanism based on priority data, persisting partition-offset data in a cache pool to an index table according to the position index cache mechanism, and performing priority consumption of high-priority messages based on persistent priority indexes, including setting a consumption control program architecture, performing priority consumption on the high-priority messages, and performing priority consumption on the high-priority messages based on the persistent priority indexes. Isolating the consumption queue and introducing a message de-duplication operation; according to the method, full-link performance adjustment and optimization are carried out on the premise of preferential consumption of high-priority messages, service scene extension is carried out based on the adjusted and optimized full links, the messages are pulled directionally according to the priorities through a consumption control program, the disorder of parallel consumption of Kafka partitions is broken through, the high-priority messages can directly jump to corresponding positions to be processed preferentially, and the service scene extension efficiency is improved. The consumption delay is obviously shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of message queues, and in particular to a method and system for controllable optimization of messages based on Kafka. Background Art

[0002] As a high-throughput distributed publish-subscribe message system, Kafka achieves high concurrency and traffic peak shaving through its partition mechanism and is widely used in scenarios such as log transmission, e-commerce data collection, and system monitoring. In its core architecture, producers send messages to different partitions of a topic, and consumers pull messages from the partitions through a consumer group. The partition mechanism allows data to be locally ordered within a single partition but is overall unordered. This feature can play a high-performance advantage in scenarios without global sorting requirements.

[0003] In the traditional Kafka architecture, each partition consumes in parallel and shares resources, lacking a message priority differentiation mechanism. When a partition contains important data, its acquisition timeliness may be squeezed by low-priority data from other partitions, resulting in high-priority messages not being processed in a timely manner. In addition, the native design of Kafka does not make semantic-level distinctions about the importance of messages, and all messages are equally processed at the consumer end. However, in actual business, it is often necessary to divide the priority of messages. The traditional architecture needs to rely on the business layer to implement additional sorting logic, increasing system complexity and latency. For example, in an e-commerce system, if user payment messages cannot be processed prior to browsing records, it may lead to a lag in transaction status updates. Finally, the high-throughput feature of Kafka relies on parallel processing of partitions, and a global sorting or priority mechanism may introduce additional coordination overhead, resulting in performance degradation. Existing solutions are difficult to achieve a balance between message priority control and ordered consumption without significantly sacrificing throughput. At present, a method and system for controllable optimization of messages based on Kafka are needed. Summary of the Invention

[0004] In order to solve the problems of performance and sorting contradictions at the traditional Kafka architecture level, priority missing at the business level, and disorder bottleneck at the performance level, the present invention provides a method and system for controllable optimization of messages based on Kafka.

[0005] In the first aspect, a method for controllable optimization of messages based on Kafka provided by the present invention adopts the following technical solutions: A method for controllable optimization of messages based on Kafka includes: Obtain data attributes and calculate data priority values according to the data attributes; Build a location index cache mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain; Persist the partition-offset data in the cache pool to the index table according to the location index caching mechanism, including defining the index table structure and setting the batch writing strategy; Perform preferential consumption of high-priority messages based on the persisted priority index, including setting the architecture of the consumption control program and isolating the consumption queue and introducing message deduplication operations; Perform full-link performance tuning on the premise of preferential consumption of high-priority messages, including tuning the cache pool parameters; Expand the business scenario based on the tuned full link, including using KafkaMirrorMaker to achieve cross-data center synchronization of the priority index table.

[0006] Furthermore, calculating the data priority value according to the data attribute includes extracting the data source, data classification, data type, data size, and data timeliness of the message data as data attributes, calculating the priority value using weighted average, and finally performing level mapping on the priority value to obtain the numerical priority level. The priority value calculation formula is: , where represents the weight coefficient of the i-th attribute, represents the score of the i-th attribute.

[0007] Furthermore, building the location index caching mechanism based on the priority data includes adding a priority processing interceptor to the interceptor chain of the Kafka producer. The interceptor calls the priority judgment program to assign a priority level to the message. In the callback function when the message is sent successfully, obtain the RecordMetadata object containing the partition and offset, and add a priority level field to it to form extended metadata. Then build a hierarchical cache pool to store the extended metadata.

[0008] Furthermore, building the hierarchical cache pool to store the extended metadata includes building an A-Cache module and a B-Cache module. Among them, the A-Cache module pre-allocates fixed-size memory blocks, and each block stores partition-offset data units of the same priority level. The capacity of a single partition-offset data unit is N records, and a lock-free queue is used to achieve fast writing. The B-Cache module dynamically allocates memory blocks to store large data units exceeding the capacity of the A-Cache, and uses the LRU algorithm to eliminate cold data. When the capacity of the partition-offset data unit in the A-Cache module is full, trigger batch writing to the data priority index table, and the number of records written each time is N.

[0009] Furthermore, a new priority processing interceptor is added to the interceptor chain of the Kafka producer, including implementing the onSend method through the Kafka interceptor interface ProducerInterceptor, injecting the priority level label into the message header before message serialization, and in the callback method, parsing the metadata object and extracting the triple including the partition number, unique offset and timestamp.

[0010] Furthermore, the priority consumption of high-priority messages based on the persistent priority index includes building an independent consumption control program, polling the data priority index table from high to low priority, extracting the partition number, offset and priority level of unconsumed messages, and pulling messages through Kafka's seek (P, O) interface to achieve priority-driven consumption order control, allocating independent thread pools for different priority levels, and then adopting a priority preemption strategy. When the high-priority queue has a task, the low-priority thread suspends consumption, and adds a unique identifier in the message header to perform message deduplication operations.

[0011] Furthermore, the full-link performance tuning is performed under the premise of priority consumption of high-priority messages, including dynamic tuning of cache pool parameters, consumer-side performance optimization, priority preemption and resource allocation. The cache pool parameters are dynamically adjusted, and the A-Cache module capacity is calculated according to the peak service message volume, and the maximum capacity of the B-Cache module is set to M. When the cache data exceeds the set threshold, the LRU algorithm is triggered to force the elimination of cold data units. The A-Cache module capacity calculation formula is: , Among them, N is the number of records stored in a single Unit. is the expansion factor.

[0012] Furthermore, the consumer-side performance optimization includes adopting a batch pull strategy to calculate the pull interval for high-priority consumer threads, and adopting adaptive pull for low-priority consumer threads to dynamically adjust the pull amount according to the partition Lag. The calculation formula for the dynamically adjusted pull amount is: , in, It is expressed as a regulation factor. The current Lag of a partition is expressed as the number of unconsumed messages in the partition, which can be obtained in real time through Kafka AdminClient.

[0013] Further, the cross-data center synchronization of the priority index table using Kafka MirrorMaker includes constructing a synchronization link for the priority index table between the primary data center and the disaster recovery data center using the Kafka MirrorMaker component, defining a priority level mapping function using a configuration file to convert the primary center priority level to the disaster recovery center level, adopting an idempotent write mechanism, and filtering duplicate synchronization data based on the unique combined key of the index table to avoid redundant storage in a distributed environment.

[0014] In a second aspect, a message controllable optimization system based on Kafka includes: A data acquisition module, configured to: acquire data attributes and calculate data priority values according to the data attributes; A cache module, configured to: construct a location index cache mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain; An index conversion module, configured to: persist partition-offset data in the cache pool to the index table according to the location index cache mechanism, including defining the index table structure and setting a batch write strategy; A conversion module, configured to: preferentially consume high-priority messages based on the persisted priority index, including setting a consumption control program architecture and isolating the consumption queue and introducing a message deduplication operation; An optimization module, configured to: perform full-link performance tuning on the premise of preferentially consuming high-priority messages, including tuning cache pool parameters; An output module, configured to: expand the business scenario based on the tuned full link, including using Kafka MirrorMaker to implement cross-data center synchronization of the priority index table.

[0015] In summary, the present invention has the following beneficial technical effects: 1. By extracting the five major attributes of data source, classification, type, size, and timeliness and establishing a weighted average model, the present invention converts the abstract business priority requirements into computable quantitative indicators, solving the problem that the Kafka native architecture cannot recognize the importance of messages.

[0016] 2. By injecting priority tags and recording message positions in the Kafka producer interceptor, the present invention realizes the integration of "priority marking" and "storage location tracking". The hierarchical cache pool (A-Cache pre-allocates fixed memory blocks, B-Cache dynamically manages large units) combined with the batch write strategy avoids the performance loss of single data writing. While ensuring the high throughput characteristics of Kafka, it ensures the real-time recording of priority data, and the index write latency can be stabilized at the microsecond level, improving the processing efficiency and stability of the producer side.

[0017] 3. The present invention breaks through the disorder of parallel consumption of Kafka partitions by pulling messages according to priorities through a consumption control program, enabling high-priority messages to directly jump to the corresponding positions for priority processing, significantly shortening the consumption latency. The design of an independent thread pool and the priority preemption strategy (high-priority threads first obtain resources) further ensure the timeliness of important data. At the same time, the message deduplication mechanism avoids duplicate processing in a distributed environment, ensuring the accuracy of business data.

[0018] 4. The cache pool parameters are dynamically adjusted according to the business peak, combined with the LRU algorithm to eliminate cold data, avoiding memory waste and performance jitter. The consumer side adopts a differentiated pulling strategy for different priorities, reducing invalid network requests and improving the overall throughput. The full-link optimization has extremely low performance loss to Kafka natively while ensuring priority control, achieving a balance between function expansion and performance.

[0019] 5. Use KafkaMirrorMaker to realize the synchronization of the priority index table across data centers, support custom priority level mapping, meet the differentiated requirements of different business scenarios under a distributed architecture. The synchronization link optimization (high-priority independent channels, idempotent writing) ensures cross-center data consistency and timeliness, with a short disaster recovery switching latency, guaranteeing business continuity and improving the reliability and scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 FIG. is a schematic diagram of the overall process of a message controllable optimization method based on kafka according to an embodiment of the present invention.

[0021] Figure 2 FIG. is a general technical architecture diagram of a message controllable optimization method based on kafka according to an embodiment of the present invention.

[0022] Figure 3 FIG. is a logic diagram of the offset storage - high-throughput cache algorithm in a message controllable optimization method based on kafka according to an embodiment of the present invention.

[0023] Figure 4 FIG. is a logic structure diagram of data priority indexing in a message controllable optimization method based on kafka according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] The present invention will be further described in detail below with reference to the accompanying drawings.

[0025] Embodiment 1 Referring to Figure 1 , a message controllable optimization method based on kafka in this embodiment includes: Obtain data attributes and calculate data priority values according to the data attributes; Build a location index caching mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain; Persist the partition-offset data in the cache pool to the index table according to the location index caching mechanism, including defining the index table structure and setting the batch writing strategy; Prioritize the consumption of high-priority messages based on the persisted priority index, including setting the consumption control program architecture and isolating the consumption queue and introducing message deduplication operations; Perform full-link performance tuning on the premise of prioritizing the consumption of high-priority messages, including tuning the cache pool parameters; Expand the business scenario based on the tuned full-link, including using Kafka MirrorMaker to achieve cross-data center synchronization of the priority index table.

[0026] Specifically, a message controllable optimization method based on Kafka includes the following steps: S1. Obtain data attributes and calculate the data priority value according to the data attributes; As Figure 1 shown, by extracting the five major attributes of the message and constructing a weighted average model, the quantization and grading of data priority are realized. At the Kafka producer side, through a custom interceptor (ProducerInterceptor), the message metadata or payload is parsed to extract the following five major attributes: data source, data classification, data type, data size, and data timeliness. Then, each attribute is converted into a standardized score on a 0-100 scale. Among them, the data source is specifically assigned 80-100 points for the core system (such as ERP) and 0-20 points for the edge system. The data classification is specifically that the first-level permission data (such as transaction instructions) is assigned 90-100 points, and the public data is assigned less than 30 points. The data type is specifically that the structured data (such as JSON) is assigned 60-80 points, and the unstructured data is assigned 20-40 points. The data size is specifically that an inverse proportional linear mapping is adopted: , which is valid when the maximum value ≤ 5MB, and is fixed at 20 points if it exceeds. The data timeliness adopts a direct proportional linear mapping , where it is valid when the actual time limit ≤ the reference time limit.

[0027] Perform a weighted average calculation based on the standardized scores of each attribute. First, preset the default weight matrix and apply the weighted average formula to calculate the priority value: , Among them, represents the weight coefficient of the i-th attribute, represents the score of the i-th attribute, and then P is mapped to a 5-level priority level (L): , Among them, P represents the priority value. Through the onSend method of the interceptor, the level label (format: priority: L) is injected into the message header (MessageHeader) before the message is serialized to ensure that the priority information is transmitted along the entire message link.

[0028] S2, build a location index cache mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain; like Figure 2 , Figure 3 As shown in the figure, in the process of Kafka producer sending messages, a priority processing interceptor is added by implementing the ProducerInterceptor interface. The interceptor intervenes in the processing before message serialization (onSend method) and after the message is successfully sent (onAcknowledgment callback): Message header label injection: In the onSend method, the message content is parsed and the priority judgment program of the business system (such as the weighted average algorithm) is called to assign a priority level L (0-4, 0 is the highest) to the message, and the level is written to the message header (MessageHeader) in the form of the label priority: L. This label is transmitted to the Kafka cluster with the message and becomes the core identifier for subsequent processing. Extended metadata generation: After the message is successfully sent to the partition, the RecordMetadata object returned by Kafka (including partition number partition, offset offset, timestamp timestamp) is obtained through the onAcknowledgment callback, and the priority level field is added to it to form a custom PriorityRecordMetadata object, which binds the priority to the message storage location (partition + offset) to provide key data for subsequent index cache.

[0029] In the Kafka messaging system, producers need to record partition and offset information after sending messages so that consumers can pull them in a priority-oriented manner. If these metadata are directly written to the database one by one, frequent IO will cause performance bottlenecks. Therefore, this embodiment designs A-Cache and B-Cache layered cache pools to improve data processing efficiency through memory caching and batch writing mechanisms, while avoiding memory overflow.

[0030] The design and data storage mechanism of the hierarchical cache pool is based on the priority level obtained in step S1. When the A-Cache module starts, a set of fixed-size memory blocks are pre-allocated. Each block is used to store the partition-offset data unit (Unit) of the same priority. Each Unit can accommodate up to N records. The calculation formula of N is: , within each Unit, a lock-free queue (such as ConcurrentLinkedQueue) is used to achieve multi-threaded concurrent writing, avoiding performance losses caused by lock contention. When the number of records in a Unit reaches N, a batch writing operation is triggered, and N records are stored in the data priority index table at once. This design reduces the number of database I / O operations to 1 / N of single-record writing, significantly improving the persistence efficiency.

[0031] B-Cache module. For units with data volume exceeding the capacity of a single block in A-Cache, such as the metadata of large-data-volume messages, B-Cache memory blocks are dynamically allocated, and the size can be flexibly adjusted, such as 4KB - 64KB. If the metadata unit size exceeds the capacity of a single block in A-Cache, it is directly stored in B-Cache; if there is no free block in A-Cache, new data will also be stored in B-Cache. The B-Cache module uses the LRU (Least Recently Used) algorithm to maintain memory blocks. When the memory occupancy exceeds a threshold (such as 80% of the total capacity), the least recently accessed cold data unit is automatically eliminated to free up space for new data. The specific working principle of the LRU algorithm is as follows: Metadata units that have not been accessed or processed for a long time are defined as cold data. Continuing to occupy memory with such data will reduce the cache efficiency. The access order of all B-Cache data units is maintained. Each node contains a reference to the data unit and an access timestamp. The most recently accessed unit is placed at the head of the linked list, and the least recently accessed unit is at the tail of the linked list. The key is the unique identifier of the data unit (such as the combination of priority + partition + offset), and the value is a reference to the linked list node. When a data unit is accessed (such as read or write), the linked list node is located through a hash table, removed from its current position, and inserted at the head of the linked list, which is marked as the most recently accessed. When the B-Cache memory occupancy exceeds the preset threshold (such as 80% of the total capacity), an elimination mechanism is triggered, starting from the tail of the linked list and eliminating the least recently accessed nodes in turn until the memory occupancy drops below the threshold.

[0032] The collaborative scheduling system with A-Cache automatically selects the cache module according to the unit data size. If the unit size ≤ the size of a single block in A-Cache and there is a free block in A-Cache, it is stored in A-Cache; otherwise, it is stored in B-Cache. This mechanism balances the efficiency of fixed blocks and the flexibility of dynamic blocks. The capacity calculation formula of the A-Cache module is: , where N is the number of records stored in a single Unit, is the expansion coefficient, and the capacity of the B-Cache module is twice that of

[0033] The newly added PriorityRecordMetadata objects are classified by priority level L. First, attempt to store them in the Unit corresponding to the priority level in A-Cache. If the capacity of the Unit is not full, continue to append records. If it is full, trigger a batch write to the index table and reset the Unit. If the unit size exceeds the A-Cache limit or there are no free blocks in A-Cache, store them in the B-Cache module, and the LRU algorithm manages the lifecycle.

[0034] Finally, perform asynchronous backup management on the cold data after eviction. When the memory occupancy of B-Cache exceeds the threshold, the LRU algorithm evicts cold data units from the tail of the linked list, triggering the asynchronous backup process. Write the evicted data units to the local file system. The backup files are stored hierarchically according to priority and time dimensions, and the path format is: / cache_backup / priority_L / YYYYMMDD / HHMM / unit_XXX.dat. The file name contains the key information of the data unit. The backup files are stored in binary format and contain the complete information of the data unit. Finally, the asynchronous backup uses two blocking queues (backupQueue and completedQueue) to implement the producer-consumer pattern: that is, store the data units to be backed up in backupQueue and store the data units that have completed backup in completedQueue.

[0035] After that, perform cold data backup recovery and re-insertion into B-Cache operations. When a consumer queries data, if neither A-Cache nor B-Cache hits, search the backup files according to priority and offset, read the backup files and deserialize them into DataUnit objects, verify the data integrity through the CRC32 checksum. Check the memory occupancy of B-Cache before recovery. If it exceeds the threshold, first trigger a round of LRU to evict old data to make room for the newly recovered data. Generate a unique identifier for the recovered data unit. If the identifier already exists in B-Cache, first remove the old node from the linked list, insert the recovered data unit as the most recently accessed node at the head of the LRU linked list, and update the reference of the lruMap hash table to ensure priority hit during the next access. The recovered data unit participates in the LRU eviction again. If it is evicted again, repeat the backup process. After the data is successfully recovered to B-Cache, delete the corresponding backup record from the completedQueue queue.

[0036] S3. Persist the partition-offset data in the cache pool to the index table according to the location index cache mechanism, including defining the index table structure and setting the batch write policy; The index table contains fields such as priority_level, partition_id, offset, and timestamp, and a composite unique index is established on (priority_level, partition_id, offset) to ensure that the location index of each message is unique and non-repeating. Using database batch insertion techniques (such as JDBC batch processing), the Unit data in the cache pool is encapsulated into SQL batch statements (such as inserting 1024 records at a time) to reduce the number of interactions with the database. The example process is as follows: The cache pool collects full-capacity Units → generates batch SQL → calls executeBatch() to execute → clears the Units and reuses them.

[0037] Among them, the priority level is calculated through the weighted average algorithm in step S1. The partition number is the Kafka partition number where the message is stored, and the offset is the unique offset of the message within the partition, which is the location identifier provided by Kafka natively to ensure message order. The timestamp is the message production timestamp, which is used for time range queries and data timeliness management. A composite unique index is established on the (priority_level, partition_id, offset) fields to ensure that records with the same priority, partition, and offset are stored only once, avoiding index redundancy caused by repeated message sending. A general index is established on the timestamp field to accelerate the scenario of querying high-priority messages by time range.

[0038] The conditions for the cache pool to trigger batch writing are as follows: Specifically for the A-Cache module: A pre-allocated fixed-size memory block (such as 8KB / block) is used to store partition-offset data units (Units) of the same priority. When the number of records stored in a single Unit reaches the preset capacity N (such as 1024), batch writing is triggered. For example, after the Unit of level 0 messages is filled with 1024 records, the data is immediately batch-written into the index table.

[0039] For the B-Cache module: Dynamically allocated memory blocks are used to store large data units that exceed the capacity of the A-Cache (such as messages with a large space occupied by a single record). When the memory occupancy of the B-Cache exceeds the threshold (such as 80% of the total capacity) or the idle time of the data unit exceeds the preset threshold (such as 5 seconds), forced writing is triggered to release the memory space.

[0040] Then set the execution logic of batch writing, sort the Unit data in the cache pool into a batch insert data set according to the order of the index table fields. For example, convert the full-capacity Unit in A-Cache into a list containing 1024 records, each record corresponds to a row of data in the index table, and use the database's batch insert interface (such as MySQL's LOADDATAINFILE and PostgreSQL's COPY command) to submit the data set to the index table at one time. Compared with single insertion, this operation reduces the number of interactions between the database and the application from N times to 1 time, significantly reducing network latency and IO overhead. After the batch write is successful, the corresponding Unit space is cleared for new data to refill; if the write fails (such as the database connection is interrupted), the Unit data is stored back in the cache pool and waits for the next trigger condition to retry.

[0041] Then, a data consistency and disaster recovery mechanism is introduced, and database transactions are enabled during batch writing: After starting a transaction, perform a batch insert operation. If all records are successfully written, commit the transaction. If it fails midway, roll back the transaction and store the unsuccessfully written data back into the cache pool to avoid index inconsistencies caused by partial data writing. For example, if 500 out of 1024 records fail to be written, the entire batch operation will be rolled back and all data will be back into the cache pool to wait for retry.

[0042] Cache data that is not written to the index table in time (such as unfull units in A-Cache and cold data in B-Cache) is asynchronously backed up to the local file system and stored in / cache_backup / priority_0 / 20250606 / unit_001.dat partitions by priority and time. The backup file contains the binary serialization result of the data to ensure rapid recovery in case of failure.

[0043] When the system restarts, scan the local backup file and parse out the records that have not been written to the index table; obtain the current maximum offset of each partition through Kafka's AdminClient, filter out the records that have not been consumed (that is, the records whose offset is greater than the current partition offset), and re-write them into the index table in batches. This process ensures that the index data that was not persisted during the failure is not lost, and the consumed messages are not processed repeatedly. Finally, the capacity N of a single unit is dynamically adjusted according to the message characteristics of the business scenario: , in, Denotes floor division. "A-Cache single-block size" is the number of bytes of the fixed-size memory block pre-allocated for the A-Cache module. "Average size of a single record" is the average number of bytes required to store a partition-offset data record. Additionally, the index table structure and batch writing logic support mainstream relational databases, and adapt to the characteristics of different databases through the abstract data access layer: for MySQL, the parameter rewriteBatchedStatements=true is used to optimize the batch insertion performance; for Oracle, the FORALL statement is used to execute batch operations in parallel; for PostgreSQL, the PREPARE and EXECUTE statements are used to precompile batch insertion statements to reduce the syntax parsing overhead.

[0044] S4. Prioritized consumption of high-priority messages based on the persistent priority index, including setting up the architecture of the consumption control program, isolating the consumption queue, and introducing message deduplication operations. As Figure 4 shown, priority index polling: in the order from high to low priority (level 0 → level 1 → level 2 → level 3 → level 4), regularly query the records with the status of "unconsumed" in the data priority index table. For example, query the indexes of level 0 and level 1 messages every 100 milliseconds, and query low-priority messages every 500 milliseconds to ensure that high-priority data is processed first.

[0045] The directed consumption is executed through the seek(partition, offset) interface provided by Kafka to directly jump to the specified offset of the target partition to pull messages. The traditional Kafka consumption process reads in sequence by partition, while this solution obtains the exact position (partition number + offset) of the message through the index table to achieve the consumption order control of "priority first". For example, if there are offsets 1000 in partition 1 and 2000 in partition 2 of level 0 messages in the index table, the control program will preferentially pull the messages at these two positions instead of starting consumption from partition 0 in the partition order. After the consumption status is updated and the message is successfully consumed, immediately mark the status of the corresponding record in the index table as "consumed" to avoid duplicate processing.

[0046] The traditional Kafka consumption processes messages in sequence by partition, and there is no priority difference between messages in different partitions. This solution classifies messages by priority through the index table, and the index records of high-priority messages will be preferentially polled and pulled, thus breaking through the partition limit. For example, level 0 messages may be distributed in multiple partitions. The control program will uniformly collect all unconsumed level 0 indexes and pull them in sequence according to the offset to ensure that messages within the same priority are processed in partition order, and between different priorities, they are processed according to the level.

[0047] After that, a hierarchical thread pool architecture and resource allocation are carried out. The high-priority thread pool uses a fixed-size thread pool (such as 5 threads), and each thread is bound to an independent task queue (with a capacity of 100). The independent queue ensures that high-priority messages will not be blocked by low-priority tasks. For example, the task queue for level 0 messages only stores the consumption tasks corresponding to the level 0 index. The thread pool focuses on processing the tasks in this queue to avoid mixing with other priorities. The fixed number of threads is 1.5 times the number of CPU cores to improve the parallel processing ability.

[0048] The low-priority thread pool uses a dynamic thread pool (such as a cached thread pool), and multiple priorities share a task queue. The number of threads is automatically adjusted according to the task volume. For example, when there are fewer low-priority tasks, the thread pool automatically reclaims idle threads to save resources; when the tasks surge, new threads are dynamically created for processing. After that, a priority preemption strategy is set, and resource preemption is achieved through the thread scheduling mechanism at the operating system level: when there are tasks to be processed in the task queue of the high-priority thread pool, the low-priority thread actively releases the CPU time slice and pauses consumption. The specific implementation method is as follows: When the high-priority thread is executing, set the thread priority to the highest (such as Thread.MAX_PRIORITY in Java). The low-priority thread regularly checks the status of the high-priority queue during operation. If it finds that there are tasks, it actively yields the CPU through Thread.yield() until the high-priority queue is empty.

[0049] Finally, message deduplication is carried out. During the message production stage, a globally unique identifier (such as UUID) is added to each message through the producer interceptor and stored in the message header (Message Header) in the format of message-id:UUID. For example, the unique identifier of a certain message is message-id:ab12cd34-5ef6-7gh8-9ij0-klm1nop2q3r4. The consumer side maintains a local cache (such as a memory-based hash table) to record the UUIDs of the processed messages. The cache capacity is set to 100,000 entries, and the LRU (Least Recently Used) algorithm is used to eliminate old records. The cache expiration time is the maximum survival period of the message (such as 7 days) to avoid occupying memory for a long time.

[0050] After receiving the message, the consumer first extracts the UUID from the message header and checks whether it exists in the local cache: if it exists, it means that the message has been processed and is directly discarded without performing business logic processing; if it does not exist, the message is submitted to the thread pool for processing, and the UUID is stored in the cache.

[0051] When the seek(partition, offset) operation fails (e.g., the partition has been deleted, or the offset exceeds the maximum offset of the current partition), the consumption control program records an error log, skips the current index record, and continues to process the next valid record to avoid blocking the entire consumption process due to a single exception.

[0052] Turn off the auto-commit mechanism of Kafka and change it to manual commit: Only after the message is successfully processed and the index table status is updated, the current consumption offset is committed to ensure that no processed data is lost during fault recovery.

[0053] S5. On the premise of preferentially consuming high-priority messages, perform full-link performance tuning, including tuning cache pool parameters; Maintain a doubly linked list to record the access order of data units in the B-Cache. When the cache occupancy exceeds 80% of the maximum capacity, start eliminating the cold data units that have not been accessed for the longest time from the end of the list until the occupancy rate drops below 70%. If the eliminated data units have not been written to the index table, the number of messages pulled by high-priority threads (level 0, level 1) is set to messages, and the throughput is improved by reducing the interaction times between the Kafka client and the Broker. The pull interval calculation formula is: , where represents the number of messages pulled each time, and the pull operation sets a timeout of milliseconds. If the number of messages does not reach messages, it returns early. Low-priority threads use adaptive pulling. Before pulling, the Lag of the target partition is obtained through the AdminClient, and the pulling volume is calculated , and the dynamic adjustment formula is: , where represents the adjustment factor, and the current Lag of the partition represents the number of unconsumed messages in the partition, which is obtained in real time through the Kafka AdminClient. The physical meaning of Lag is that the partition Lag is the number of unconsumed messages in the Kafka partition, and the calculation formula is: , For example, if the maximum offset of a certain partition is 10,000 and the consumer has consumed up to 8,000, then Lag = 2,000 messages, which reflects the backlog degree of the partition. The amount of pull adjusted by the partition Lag, the dynamic calculation of the cache pool capacity, and the elastic allocation of CPU time slices enable the system to automatically balance performance and resource utilization during traffic fluctuations without manual intervention. Consumers pull messages by specifying the partition and offset. In this embodiment, the high-priority thread directly locates the target message in the partition through the seek(partition, offset) interface, breaking through the disorder of traditional parallel consumption of partitions. The calculation of the partition Lag is based on the difference between the maximum offset of each partition and the consumer offset, reflecting the message backlog situation of the partition, which is the key basis for the adaptive pull of low-priority threads. The Lag is calculated separately for each partition.

[0054] Partitions are the basis for Kafka to implement distributed storage and parallel processing and are set on the Broker nodes of the Kafka cluster. In this solution, partitions are not only the physical units for message storage but also the key dimension of the priority pull strategy. Through mechanisms such as partition Lag calculation and partition-directed pull, cross-partition priority consumption of high-priority messages is achieved, and at the same time, the load of each partition is balanced through adaptive pull, ultimately improving the timeliness of message processing while ensuring throughput.

[0055] When the task queue of the high-priority thread pool is backlogged beyond the threshold (such as the remaining capacity of the queue < 20%), the system reduces the CPU time slice ratio of low-priority threads through the CPU scheduler of the operating system and allocates the released time slices to high-priority threads. When low-priority threads are idle (such as when the task queue is empty), they preload some high-priority index data into the memory cache to reduce the cold start latency of high-priority tasks. A-Cache adopts a design without GC. A-Cache pre-allocates memory blocks using Direct Memory or an object pool (such as Apache Commons Pool) to avoid the pauses caused by Java heap memory allocation and garbage collection (GC). In this embodiment, Unsafe.allocateMemory() is used to directly allocate an 8KB memory block to eliminate the impact of GC on high-priority tasks. B-Cache performs generational recycling, and the capacity of A-Cache is dynamically adjusted according to the peak message volume to ensure that the index cache capacity at the producer side matches the message production rate. The LRU eviction mechanism of B-Cache avoids memory overflow and reserves space for high-priority metadata.

[0056] S6. Expand the business scenario based on the tuned full link, including using KafkaMirrorMaker to achieve cross-data center synchronization of the priority index table.

[0057] Deploy the Kafka MirrorMaker component between the primary data center and the disaster recovery data center to form a unidirectional or bidirectional synchronization link: Data outflow from the primary center: When the priority index table in the primary center changes (such as adding partition-offset records of high-priority messages), the MirrorMaker producer captures the changed data, serializes it into Kafka messages, and sends them to a dedicated Topic; Data inflow to the disaster recovery center: The MirrorMaker consumer in the disaster recovery center subscribes to this Topic, parses the message content, and writes it into the local priority index table to ensure the consistency of index data between the two locations; Synchronization granularity: Only synchronize the "unconsumed" records in the index table to avoid the ineffective transmission of consumed data and reduce network traffic.

[0058] Priority level definitions may need to be adjusted due to differences in business rules in different data centers. Define mapping rules through a configuration file to convert the priority level (L) in the primary center to the level (L') in the disaster recovery center. Data that is "relatively important (level 0)" in the primary center needs to be upgraded to "important (level 1)" in the disaster recovery center to adapt to the risk control rules at the disaster recovery end; data that is "ordinary (level 3)" in the primary center remains at the original level in the disaster recovery center.

[0059] The priority index table ensures the idempotency of cross-center synchronization through the composite unique key (priority_level, partition_id, offset): Before writing synchronized data to the disaster recovery center, first query whether there is a record with the same composite key in the local index table. If it exists, it means the data has been synchronized, and the write operation is skipped directly; if it does not exist, the insert operation is executed. This mechanism avoids duplicate data caused by network retries, cluster switches, etc. For example, when the primary center repeatedly sends the same index change, the disaster recovery center only processes it once.

[0060] By adding a database index (such as a B-tree index) to the composite unique key, the response time of the deduplication query is controlled within milliseconds to ensure that the synchronization efficiency is not affected. For the indexed data in batch synchronization, first construct a set of unique keys in memory (such as using HashSet), filter out the existing records, and then write them to the database in batches to reduce the number of disk I / O operations. Establish a two-way MirrorMaker link between the primary center and the disaster recovery center. Under normal circumstances, data is synchronously replicated from the primary center to the disaster recovery center unidirectionally. When the primary center fails, manually or automatically switch to the disaster recovery center to synchronize to the primary center. Allocate independent Kafka Topic partitions for high-priority index records, and enable message compression for low-priority index data through the partition priority of Kafka or an external traffic control tool.

[0061] Embodiment 2 The difference between this embodiment and Embodiment 1 is that this embodiment provides a message controllable optimization system based on Kafka, including: A data acquisition module, configured to: acquire data attributes and calculate data priority values according to the data attributes; A cache module, configured to: construct a location index cache mechanism based on the priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain; An index conversion module, configured to: persist the partition-offset data in the cache pool to the index table according to the location index cache mechanism, including defining the index table structure and setting the batch write policy; A conversion module, configured to: preferentially consume high-priority messages based on the persisted priority index, including setting the consumption control program architecture and isolating the consumption queue and introducing message deduplication operations; An optimization module, configured to: perform full-link performance tuning on the premise of preferentially consuming high-priority messages, including tuning the cache pool parameters; An output module, configured to: expand the business scenario based on the tuned full link, including using Kafka MirrorMaker to achieve cross-data center synchronization of the priority index table.

[0062] The above are all the preferred embodiments of the present invention, and the protection scope of the present invention is not limited accordingly. Therefore, all equivalent changes made according to the structure, shape, and principle of the present invention should be covered within the protection scope of the present invention.

Claims

1. A message controllable optimization method based on Kafka, characterized in that Including: Obtain data attributes and calculate data priority values according to the data attributes; Build a location index caching mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain; Persist the partition-offset data in the cache pool to the index table according to the location index caching mechanism, including defining the index table structure and setting the batch writing strategy; Perform preferential consumption of high-priority messages based on the persisted priority index, including setting the consumption control program architecture and isolating the consumption queue and introducing message deduplication operations; Perform full-link performance tuning on the premise of preferential consumption of high-priority messages, including tuning the cache pool parameters; Expand the business scenario based on the tuned full link, including using KafkaMirrorMaker to achieve cross-data center synchronization of the priority index table.

2. The message controllable optimization method based on Kafka according to claim 1, wherein The calculating the data priority value according to the data attributes includes extracting the data source, data classification, data type, data size, and data timeliness of the message data as data attributes, calculating the priority value using weighted average, and finally performing level mapping on the priority value to obtain the numerical priority level. The priority value calculation formula is: ， Among them, represents the weight coefficient of the i-th attribute, represents the score value of the i-th attribute.

3. The message controllable optimization method based on Kafka according to claim 1, wherein The building the location index caching mechanism based on priority data includes adding a priority processing interceptor to the interceptor chain of the Kafka producer. The interceptor calls the priority judgment program to assign a priority level to the message. In the callback function when the message is sent successfully, obtain the RecordMetadata object containing the partition and offset, and add a priority level field to it to form extended metadata, and then build a hierarchical cache pool to store the extended metadata.

4. The message controllable optimization method based on Kafka according to claim 3, wherein The building the hierarchical cache pool to store the extended metadata includes building an A-Cache module and a B-Cache module. Among them, the A-Cache module pre-allocates fixed-size memory blocks, and each block stores partition-offset data units of the same priority level. The capacity of a single partition-offset data unit is N records, and a lock-free queue is used to achieve fast writing. The B-Cache module dynamically allocates memory blocks to store large data units exceeding the capacity of the A-Cache, and uses the LRU algorithm to eliminate cold data. When the capacity of the partition-offset data unit in the A-Cache module is full, it triggers batch writing to the data priority index table, and the number of records written each time is N.

5. A message controllable optimization method based on Kafka according to claim 4, characterized in that The adding a priority processing interceptor to the interceptor chain of the Kafka producer includes implementing the onSend method through the Kafka interceptor interface ProducerInterceptor, injecting a priority level tag into the message header before message serialization, and in the callback method, parsing the metadata object and extracting a triple including the partition number, unique offset, and timestamp.

6. A method for controllable optimization of messages based on Kafka according to claim 1, characterized in that The priority consumption of high-priority messages based on the persistent priority index includes building an independent consumption control program, polling the data priority index table from high to low priority, extracting the partition number, offset and priority level of the unconsumed message, pulling messages through the seek (P, O) interface of Kafka, realizing priority-driven consumption order control, allocating independent thread pools for different priority levels, and then adopting a priority preemption strategy. When the high-priority queue has a task, the low-priority thread suspends consumption, and adds a unique identifier in the message header to perform message deduplication.

7. A message controllable optimization method based on Kafka according to claim 1, characterized in that The full-link performance tuning is performed under the premise of preferential consumption of high-priority messages, including dynamic tuning of cache pool parameters, consumer-side performance optimization, priority preemption and resource allocation. The cache pool parameters are dynamically adjusted, and the A-Cache module capacity is calculated according to the peak service message volume. The maximum capacity of the B-Cache module is set to M. When the cache data exceeds the set threshold, the LRU algorithm is triggered to force the elimination of cold data units. The A-Cache module capacity calculation formula is: ， where N is the number of records stored in a single Unit, is the expansion coefficient.

8. A method for controllable optimization of messages based on Kafka according to claim 7, characterized in that The consumer-side performance optimization includes adopting a batch pull strategy to calculate the pull interval for high-priority consumer threads, and adopting adaptive pull for low-priority consumer threads to dynamically adjust the pull amount according to the partition Lag. The calculation formula for the dynamically adjusted pull amount is: ， Among them, is expressed as a regulation factor, and the current Lag of the partition is expressed as the number of unconsumed messages in the partition, which is obtained in real time through the KafkaAdminClient.

9. A method for controllable optimization of messages based on Kafka according to claim 1, characterized in that The cross-data center synchronization of the priority index table is achieved by using Kafka MirrorMaker, including using the Kafka MirrorMaker component to build a priority index table synchronization link between the main data center and the disaster recovery data center, using a configuration file to define a priority level mapping function, converting the main center priority level into the disaster recovery center level, using an idempotent write mechanism, and filtering duplicate synchronization data based on the unique joint key of the index table to avoid redundant storage in a distributed environment.

10. A message controllable optimization system based on Kafka, which executes the method described in claim 1, characterized in that include: The data acquisition module is configured to: acquire data attributes and calculate data priority values according to the data attributes; The cache module is configured to: build a location index cache mechanism based on priority data, including adding a priority processing interceptor to the Kafka producer interceptor chain; The index conversion module is configured to: persist the partition-offset data in the cache pool to the index table according to the position index cache mechanism, including defining the index table structure and setting the batch write strategy; The conversion module is configured to: prioritize the consumption of high-priority messages based on the persistent priority index, including setting the consumption control program architecture, isolating the consumption queues and introducing message deduplication operations; The optimization module is configured to perform full-link performance tuning, including tuning of cache pool parameters, on the premise of giving priority to the consumption of high-priority messages. The output module is configured to expand business scenarios based on the optimized full link, including using KafkaMirrorMaker to achieve cross-data center synchronization of priority index tables.

Images