Out-of-order processing method and device for multi-association real-time data stream, medium and equipment

Abstract

The invention provides an out-of-order processing method and device for a multi-association real-time data stream, a medium and equipment. The method comprises the steps: monitoring the log changes of a main table and / or an association table in a database in real time; when the log changes, writing the main table change record and the association table change record into corresponding topics of the message queue in parallel in a multi-partition mode; acquiring multi-partition data streams of the main table and the association table; loading a service primary key of the primary table to a cache region; obtaining a service foreign key, and judging whether a corresponding service primary key can be obtained from the cache region or not; if the service main key can be obtained, combining the multi-partition data streams of the main table and the association table into independent partition data streams, grouping the independent partition data streams according to specified fields, sorting the grouped data streams, consuming the grouped data streams, and executing service logic processing on the consumed data flow. According to the method, disorder in a high-concurrency multi-association data stream real-time scene can be solved.

Description

technical field [0001] The invention relates to the field, in particular to a method, device, medium and equipment for out-of-order processing of multi-associated real-time data streams. Background technique [0002] In the era of big data, with the continuous increase of data volume, various types of data and more and more real-time big data applications, Hadoop, as an offline high-throughput and low-response framework, can no longer meet such requirements, so real-time computing framework Micro-batch or stream processing to reduce data latency to ensure that data can be presented to users efficiently. [0003] When using the real-time computing framework to process data, the specific implementation is to first store the external data in the message queue, use the real-time framework to consume the data under the corresponding topic in real time and process the business logic, so as to reduce the data delay. [0004] The inventor finds in the process of realizing the prese...

AI Technical Summary

Problems solved by technology

[0005] When the real-time computing framework is used to process the data flow, it is faced with the problems of data delay, loss, and duplication. Therefore, it is necessary to store the delayed data and the offset of consumption in the buffer area to ensure the accuracy of the data, but in improving the task It is difficult to guarantee the sequence problem in the case of multiple partitions with a high degree of parallelism, so in some business scenarios that need to guarantee the sequence, it will lead to deviations

Images