Kafka-based ordered processing method and system for same data during production and consumption

Abstract

The invention provides a kafka-based ordered processing method and system for same data during production and consumption. The method comprises a step of orderly pushing by a kafka production end anda step of orderly consuming by a kafka consumption end. The step of orderly pushing by the kafka production end comprises the following sub-steps: S1, submitting messages to an Oracle database by a plurality of application ends according to affairs, monitoring message changes of the Oracle database by an OGG synchronization program, and allocating different processes on the OGG synchronization program to process the monitored message queues according to the affairs; s2, rewriting the processing logic of the OGG to obtain a section partition number N corresponding to the service primary key; s3, rewriting the sent custom partition, and sending the N value calculated by the same service primary key to the corresponding Nth section partition; s4, the same OGG thread processes the same transaction, and data in one transaction is sent to a fixed section partition; the method has the beneficial effects that the same service main keys can be ensured to be pushed in order, the consumption endconsumes in order, and the data is ensured to be consumed in order.

Description

technical field [0001] The present invention relates to the technical field of data processing, and more specifically, the present invention relates to a kafka-based sequential processing method and system for producing and consuming the same data. Background technique [0002] Kafka is a distributed streaming platform that can use data when needed by decoupling data streams. In cases where slow integrations are not required, Kafka can reduce latency (or the time it takes each data point to load) to only 10 milliseconds (about 10 times or more compared to other integrations), thereby keeping it low. Latency and high throughput performance, that is, Kafka can provide data in real time. [0003] Kafka also has high horizontal scalability and supports distributed deployment. Each message has a topic, and each topic corresponds to multiple partitions. The consumer uses multi-threaded consumption. After any message of the same Partition is processed and submitted to Offset, a u...

AI Technical Summary

Problems solved by technology

After any message of the same Partition is processed and submitted to Offset, a unique message under the Partition can be determined, and the order can be guaranteed under the Partition, but the order of the messages cannot be guaranteed for the Topic

[0004] The addition, deletion, and modification of the same business primary key will generate multiple pieces of data on Kafka, and these data will be randomly (Kafka’s default mechanism) sent to different Partition partitions, and will eventually be consumed by different threads, and the efficiency of thread consumption is often impossible Control will cause the messages in different Partitions to be consumed out of order. It may be that the production end inserts first, updates and deletes last, and the final consumption order becomes delete first, then update and insert, which will obviously produce follow-up. Inconsistency of source data leads to unexpected serious consequences in business processing

Images