A method for processing data with specified push time when consuming Kafka

Abstract

The invention discloses a method for processing data of a specified push time of consumption by Kafka, and relates to the technical field of data processing; the method comprises the following steps: S1, obtaining an offset of a specified consumption time, comprising the steps of finding a timestamp of Kafka, and obtaining the offset of the specified consumption time through the timestamp of Kafka; S2, setting a specified offset for Kafka; calling Kafka Consumer in Kafka API, and adopting a seek method to jump to a specified offset of a specified Partition for consumption; S3, starting a Kafka consumption program, and making the set time consistent with the timestamp corresponding to the consumption data; the invention has the beneficial effects that the method reduces the processing of duplicate data, ensures data quality, and reduces the processing capacity of other consumption applications in the same network segment affected by excessive bandwidth occupation.

Description

Technical Field

[0001] The present invention relates to the technical field of message processing, and more specifically, to a method for processing Kafka consumption specified push time data. Background Art

[0002] Kafka is a distributed publish-subscribe messaging system with features such as high throughput, persistence, replica set mechanism, and distributed support for horizontal expansion. The design of producers and consumers is completely decoupled, traffic peaks can be cut, and message processing can be asynchronous. Now it has been used by many companies as various types of data pipelines and messaging systems.

[0003] In order to better illustrate and understand the technical solution of the present invention, the basic concepts of Kafka are introduced as follows:

[0004] 1. Producers and Consumers

[0005] For Kafka, there are two basic types of clients, including: Producer and Consumer. Producers (also called publishers) create messages, while consumers (also called subscribers) are responsible for consuming messages.

[0006] 2. Topics and Partitions

[0007] In Kafka, messages are classified by topic, and each topic corresponds to a "message queue", which is similar to a table in a database. However, if all messages of the same type are stuffed into a "central" queue, it will inevitably lack scalability. Whether the number of producers / consumers or the number of messages increases, the system's performance or storage may be exhausted. To address this problem, this solution introduces the concept of partition to achieve horizontal expansion.

[0008] 3. Broker and Cluster

[0009] A Kafka server is also called a Broker. It receives messages sent by producers and stores them on disk. The Broker also serves consumers' requests to pull partition messages and returns the messages that have been submitted so far. Using specific machine hardware, a Broker can process thousands of partitions and millions of messages per second. Several Brokers form a cluster, in which a Broker in the cluster becomes a cluster controller, which is responsible for managing the cluster, including assigning partitions to Brokers, monitoring Broker failures, etc. In a cluster, a partition is managed by a Broker, which is also called the Leader of this partition. Of course, a partition can be replicated to multiple Brokers to achieve redundancy, so that when a Broker fails, its partition can be reallocated to other Brokers to be responsible.

[0010] In actual usage scenarios, Kafka is used in conjunction with other technologies to ensure that data quality is maintained when general problems occur in the Kafka cluster. However, when restarting the Kafka consumer application, the two default consumption mechanisms have their drawbacks: (1) Consuming from scratch results in excessive repeated consumption of data. When the consumption volume is huge and the consumption time is limited, it relies too much on traffic bandwidth and consumption speed. (2) Consuming from the latest data may result in missing data, affecting the final data quality. Summary of the invention

[0011] In order to overcome the shortcomings of the prior art, the present invention provides a Kafka consumption specified push time data processing method, which reduces the processing of duplicate data, ensures data quality, and reduces the impact of excessive bandwidth occupancy on the processing capacity of other consumer applications in the same network segment.

[0012] The technical solution adopted by the present invention to solve the technical problem is: a method for processing data of a specified push time consumed by Kafka, characterized in that:

[0013] S1. Obtaining the offset of the specified consumption time, including the steps of finding the timestamp of Kafka and obtaining the offset of the specified consumption time through the timestamp of Kafka;

[0014] S2. Set a specified offset for Kafka. Call the Kafka Consumer in the Kafka API and use the seek method to jump to the specified offset of the specified Partition for consumption.

[0015] S3. Start the Kafka consumer program and make the set time consistent with the timestamp corresponding to the consumption data.

[0016] Furthermore, in step S1, the steps of processing timestamps inside Kafka are as follows:

[0017] S11, the timestamp of the message written by the producer is sent to the leader broker;

[0018] S12. Timestamps are divided into LogAppend Time and Create Time by type. LogAppend Time indicates the time when the producer creates the message, and Create Time is the time when the leader broker writes the message to the log.

[0019] S13. If the timestamp is LogAppend Time, determine whether the message is compressed. If it is a compressed message, multiple messages are packaged to form an inner layer, and the outer timestamp is the unified timestamp of the batch of packaged messages. The outer timestamp is overwritten with the current time. If it is an uncompressed message, each message is independent, and the timestamp of each message is overwritten with the current time, and then enter step S14;

[0020] If the timestamp is Create Time, it is determined that the time difference between the message generation and reception is less than the set threshold, and the message is written, and then the process proceeds to step S14;

[0021] S14. Write the timestamp to the Active log segment.

[0022] Furthermore, in step S1, obtaining the offset of the specified consumption time through the timestamp of Kafka includes the following steps:

[0023] S15, putting the file name and the last time of the file into a map;

[0024] S16. When the user specifies a time t0, find the last time t1 earlier than t0 in the map and return the file name, that is, the first offset of the file;

[0025] S17. Return the offsets of all partitions, and perform steps S15-S16 for all partitions.

[0026] Furthermore, the time uses a 13-bit Unix timestamp.

[0027] Furthermore, in the step S2, when calling the seek method, it is necessary to first obtain the partition information, and the partition information is obtained by the poll method;

[0028] If there is no partition information when the seek method is called, an exception will be thrown.

[0029] Furthermore, the steps to obtain the offset through the poll method are as follows:

[0030] S21, obtain partition data through the poll method;

[0031] S22. Obtain the offset corresponding to the specified time of all partitions;

[0032] S23. Set partition offset;

[0033] S24, determine whether there are any unconfigured partitions, if yes, return to step S23, if no, proceed to the next step;

[0034] S25. Submit the offset.

[0035] The beneficial effects of the present invention are: first, the method of the present invention can reduce the processing of duplicate data, and the data processing speed is not likely to become a bottleneck; second, data loss is not likely to occur, and data quality is guaranteed; third, unnecessary duplicate data processing is removed, and the processing capacity of other consumer applications in the same network segment affected by excessive bandwidth occupancy is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The present invention is a flowchart of a method for processing data of a Kafka consumption specified push time.

[0037] Figure 2 A schematic diagram of the steps of processing timestamps within Kafka of a method for processing data of a specified push time for Kafka consumption according to the present invention.

[0038] Figure 3 A schematic diagram of the steps of obtaining an offset through a poll method in a method for processing data of a Kafka consumption specified push time according to the present invention.

[0039] Figure 4 , Figure 5 This is a diagram of an embodiment of the time set in the present invention and the timestamp corresponding to the consumption data. DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0041] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0042] Reference Figure 1 As shown, the present invention discloses a method for processing data of a specified push time for consuming Kafka. By this method, the disadvantages of consuming data from the beginning and consuming from the latest data can be avoided, and messages of a specified time can be consumed. Specifically, in this embodiment, the data processing method includes the following steps:

[0043] S1. Obtaining the offset of the specified consumption time, including the steps of finding the timestamp of Kafka and obtaining the offset of the specified consumption time through the timestamp of Kafka;

[0044] We explain in detail the steps of processing timestamps and obtaining offsets in Kafka. Figure 2 As shown, in step S1, the steps for processing timestamps inside Kafka are as follows, including steps S11-S14:

[0045] S11, the timestamp of the message written by the producer is sent to the leader broker;

[0046] S12. Timestamps are divided into LogAppend Time and Create Time by type. LogAppend Time indicates the time when the producer creates the message, and Create Time is the time when the leader broker writes the message to the log.

[0047] S13. If the timestamp is LogAppend Time, determine whether the message is compressed. If it is a compressed message, multiple messages are packaged to form an inner layer, and the outer timestamp is the unified timestamp of the batch of packaged messages. The outer timestamp is overwritten with the current time. If it is an uncompressed message, each message is independent, and the timestamp of each message is overwritten with the current time, and then enter step S14;

[0048] If the timestamp is Create Time, it is determined that the time difference between the message generation and reception is less than the set threshold, and the message is written, and then the process proceeds to step S14;

[0049] S14. Write the timestamp to the Active log segment.

[0050] Leader / follower can read the timestamp from the Active log segment and build the index file.

[0051] If the timestamp is Create Time, it is necessary to determine the time difference between the generation and reception of the message. The threshold referred to here can be defined. For example, when the time difference between the generation and reception of the message is greater than 3 minutes, the message can be not created, and writing is directly rejected and the message is directly discarded.

[0052] Based on this, we can see that Kafka will keep the current timestamp when writing data to the file, so the corresponding offset can be found through the timestamp. Getting the offset of the specified consumption time through the timestamp of Kafka includes steps S15-S17:

[0053] S15, put the file name and the last time of the file into a map; the time uses a 13-bit Unix timestamp;

[0054] S16. When the user specifies a time t0, find the last time t1 earlier than t0 in the map and return the file name, that is, the first offset of the file;

[0055] S17. Return the offsets of all partitions, and perform steps S15-S16 for all partitions.

[0056] Further, combined with Figure 2 As shown, the content of step S2 is as follows:

[0057] S2. Set a specified offset for Kafka. Call the Kafka Consumer in the Kafka API and use the seek method to jump to the specified offset of the specified Partition for consumption.

[0058] In step S2, when calling the seek method, it is necessary to first obtain the partition information, which is obtained through the poll method; if there is no partition information when calling the seek method, an exception will be thrown. Figure 3 As shown, the steps to obtain the offset through the poll method are as follows:

[0059] S21. Obtain partition data through the poll method;

[0060] S22. Obtain the offset corresponding to the specified time of all partitions;

[0061] S23. Set partition offset;

[0062] S24, determine whether there are any unconfigured partitions, if yes, return to step S23, if no, proceed to the next step;

[0063] S25. Submit the offset.

[0064] S3. Start the Kafka consumer program and make the set time consistent with the timestamp corresponding to the consumption data. Figure 4 , Figure 5 As shown, this is an embodiment of the timestamp corresponding to the set time and the consumption data. It can be seen that the set time is consistent with the consumption time.

[0065] The above method of the present invention has at least the following advantages compared to the two default consumption mechanisms in the prior art when restarting the Kafka consumer application: First, the method of the present invention can reduce the processing of duplicate data, and the data processing speed is not likely to become a bottleneck; second, data loss is not likely to occur, and data quality is guaranteed; third, unnecessary duplicate data processing is removed, and the processing capacity of other consumer applications in the same network segment affected by excessive bandwidth occupancy is reduced.

[0066] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for processing data of a specified push time for consumption by Kafka, characterized in that: S1. Obtaining the offset of the specified consumption time, including the steps of finding the timestamp of Kafka and obtaining the offset of the specified consumption time through the timestamp of Kafka; In step S1, the steps for processing timestamps within Kafka are as follows: S11, the timestamp of the message written by the producer is sent to the leader broker; S12. Timestamps are divided into LogAppend Time and Create Time by type. LogAppend Time indicates the time when the producer creates the message, and Create Time is the time when the leader broker writes the message to the log. S13. If the timestamp is LogAppend Time, determine whether the message is compressed. If it is a compressed message, multiple messages are packaged to form an inner layer, and the outer timestamp is the unified timestamp of the batch of packaged messages. The outer timestamp is overwritten with the current time. If it is an uncompressed message, each message is independent, and the timestamp of each message is overwritten with the current time, and then enter step S14; If the timestamp is Create Time, it is determined that the time difference between the message generation and reception is less than the set threshold, and the message is written, and then the process proceeds to step S14; S14, write the timestamp to the Active log segment; In step S1, obtaining the offset of the specified consumption time through the Kafka timestamp includes the following steps: S15, put the file name and the last time of the file into a map, where the time uses a 13-bit Unix timestamp; S16. When the user specifies a time t0, find the last time t1 earlier than t0 in the map and return the file name, that is, the first offset of the file; S17. Return the offsets of all partitions, and perform steps S15-S16 for all partitions. S2. Set a specified offset for Kafka. Call Kafka Consumer in the Kafka API and use the seek method to jump to the specified offset of the specified Partition for consumption. In step S2, when calling the seek method, it is necessary to first obtain the partition information, which is obtained through the poll method; If there is no partition information when the seek method is called, an exception will be thrown; The steps to obtain the offset through the poll method are as follows: S21. Obtain partition data through the poll method; S22. Obtain the offset corresponding to the specified time of all partitions; S23. Set partition offset; S24, determine whether there are any unconfigured partitions, if yes, return to step S23, if no, proceed to the next step; S25, submit offset; S3. Start the Kafka consumer program and make the set time consistent with the timestamp corresponding to the consumption data.

Images