Concurrent consumption method, apparatus and device based on Kafka and medium

Abstract

This application discloses a concurrent consumption method, apparatus, device, and medium based on Kafka, relating to the field of computer technology. The method includes: in response to detecting that the pull processing thread has completed message pulling, grouping messages according to the topic and its partitions, and sending the messages to the message queues of the corresponding groups of topics; each topic's message queue sends the corresponding messages to the corresponding topic processing thread, and the topic processing thread submits the corresponding messages to the partitioned thread pool for concurrent consumption; in response to detecting that the consumption status of all messages before any message to be committed is consumed, the corresponding topic processing thread commits the offsets of all messages before the current message to be committed to the commit queue, and the commit queue returns the commit messages to the pull processing thread. This application implements single-threaded asynchronous consumption and multi-threaded concurrent consumption of Kafka.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a concurrent consumption method, apparatus, device, and medium based on Kafka. Background Technology

[0002] Kafka is an open-source stream processing platform developed by the Apache Software Foundation, written in Scala and Java. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action streams from consumers on a website. This data is typically handled through log processing and log aggregation due to throughput requirements. Kafka aims to unify online and offline message processing through Hadoop's parallel loading mechanism, and also to provide real-time messaging through clusters.

[0003] Message middleware is primarily used in distributed systems requiring reliable data transmission. In systems employing message middleware mechanisms, different objects interact by exchanging messages. The message sender sends a message to the message server, which stores the message in its system and forwards it to the receiver when appropriate. Message middleware enables communication between different platforms and is often used to shield the characteristics of various platforms and protocols, enabling synchronous or asynchronous collaboration between applications.

[0004] Kafka's production, storage, and consumption are all based on a streaming architecture. Messages are stored linearly in the storage system and sent linearly to consumers. Currently, consumers pull a batch of messages, such as messages0 to 10. Users start consuming messages sequentially from message0 and synchronously submit a consumption identifier (ACK) to the server. When the program restarts or recovers from an anomaly, consumption starts pulling messages from the last ACK to avoid message loss.

[0005] Current Kafka clients have the following drawbacks:

[0006] 1. The consumer is single-threaded, and message retrieval (Poll), consumption (Process), and message acknowledgment (ACK) are all processed synchronously in a single thread, resulting in poor processing performance;

[0007] 2. Kafka messages need to be consumed sequentially, and the ACK is submitted at the last consumption point. When users want to implement multi-threaded consumption themselves, the management of ACKs becomes very difficult.

[0008] Therefore, how to achieve single-threaded asynchronous consumption and multi-threaded concurrent consumption in Kafka is a technical problem that urgently needs to be solved. Summary of the Invention

[0009] To address at least one of the problems mentioned in the background art, this application provides a concurrent consumption method, apparatus, device, and medium based on Kafka, which can realize single-threaded asynchronous consumption and multi-threaded concurrent consumption of Kafka.

[0010] The specific technical solutions provided in this application are as follows:

[0011] Firstly, a concurrent consumption method based on Kafka is provided, including:

[0012] In response to the detection that the message retrieval processing thread has completed message retrieval, the message is sent to the message queue of the topic corresponding to the topic group according to the topic and the partition group of the topic;

[0013] The message queue of each topic sends the corresponding message to the corresponding topic processing thread, and the topic processing thread submits the corresponding message to the partition thread pool for concurrent consumption;

[0014] In response to the detection that all messages preceding any message to be committed have been consumed, the corresponding topic processing thread submits the offsets of all messages preceding the current message to be committed to the commit queue, and the commit queue returns the information to the pull processing thread.

[0015] Furthermore, the method also includes:

[0016] In response to the detection that any message to be committed and all messages preceding it in the order of the message to be committed have been consumed, the corresponding topic processing thread submits the offset of the current message to be committed and all messages preceding it in the order of the current message to be committed to the commit queue, and the commit queue returns it to the pull processing thread.

[0017] Furthermore, the method also includes:

[0018] A bitset of bits corresponding to the number of messages to be submitted is generated according to the batch submission parameter configuration. Each status bit in the bitset is used to describe the consumption status of the messages to be submitted in the corresponding order.

[0019] In response to detecting a change in the consumption status of the message to be submitted, the status bit corresponding to the message to be submitted is updated.

[0020] Furthermore, the status bit includes at least one of 0 and 1, wherein 0 is used to describe the consumption status as unconsumed and 1 is used to describe the consumption status as consumed.

[0021] Furthermore, in response to detecting that all messages preceding any message in the commit order have been consumed, the corresponding topic processing thread submits the offsets of all messages preceding the current commit order to the commit queue, and the commit queue returns the information to the pull processing thread, including:

[0022] In response to the detection that all messages preceding any message to be committed have a status bit of 1 in the bitset, the topic processing thread submits the offset of all messages preceding the current message to be committed to the commit queue, and the commit queue returns the information to the pull processing thread.

[0023] In response to detecting that the consumption status of any message to be committed and all messages preceding the message to be committed are consumed, the corresponding topic processing thread submits the offset of the currently pending message and all messages preceding the message to be committed to the commit queue, and the commit queue returns the information to the pull processing thread, including:

[0024] In response to the detection that the status bits in the bitset corresponding to any message to be committed and all messages preceding the message to be committed are 1, the topic processing thread submits the offset corresponding to the current message to be committed and all messages preceding the message to be committed to the commit queue, and the commit queue returns it to the pull processing thread.

[0025] Furthermore, the method also includes:

[0026] In response to detecting that the current time of the commit queue is more than a preset time threshold from the previous received offset time, the status bit information of the bitset is obtained;

[0027] Logs are generated based on the status information of the bitset, and the logs are returned to the fetch processing thread.

[0028] Furthermore, the batch submission parameter configuration includes at least one of the following: the number of messages submitted per batch, the number of bits in the bitset, and a preset time threshold.

[0029] Secondly, a concurrent consumption device based on Kafka is provided, the device comprising:

[0030] The message retrieval and distribution module is used to respond to the detection that the retrieval processing thread has completed message retrieval, and to send the message to the message queue of the topic in the corresponding group according to the topic and the partition group of the topic;

[0031] The message processing module is used to send the corresponding message from the message queue of each topic to the corresponding topic processing thread, and the topic processing thread submits the corresponding message to the partition thread pool for concurrent consumption.

[0032] The offset submission module is used to respond to the detection that all messages preceding any message to be submitted in the order of messages to be submitted have been consumed. The corresponding topic processing thread submits the offset of all messages preceding the current message to be submitted to the commit queue, and the commit queue returns the message to the pull processing thread.

[0033] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the Kafka-based concurrent consumption method.

[0034] Fourthly, a computer-readable storage medium is provided, storing computer-executable instructions for executing the Kafka-based concurrent consumption method.

[0035] The embodiments of this application have the following beneficial effects:

[0036] This application provides a Kafka-based concurrent consumption method, apparatus, device, and medium that can isolate the poll thread and the process thread, decouple the poll thread and the process thread, and achieve asynchronous partition consumption, thereby improving concurrency. Simultaneously, it sends corresponding messages to the corresponding topic processing thread through the topic's message queue, and the topic processing thread (topic_processor) submits the corresponding messages to the partition thread pool (partition_poll) for concurrent consumption. By checking that all messages before the current message to be submitted are consumed before executing offset commit, it avoids the loss of unconsumed messages, further improving concurrency, increasing consumption performance, and achieving multi-threaded concurrent consumption. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This document shows the overall flowchart of the concurrent consumption method based on Kafka provided in an embodiment of this application;

[0039] Figure 2 A detailed flowchart of a Kafka-based concurrent consumption method according to an embodiment of this application is shown;

[0040] Figure 3 A framework diagram of a Kafka-based concurrent consumption method according to an embodiment of this application is shown;

[0041] Figure 4 This diagram illustrates the message queue consumption status according to an embodiment of this application.

[0042] Figure 5 This application illustrates a corresponding embodiment. Figure 4 A schematic diagram of the bieset state bits;

[0043] Figure 6 This illustration shows a schematic diagram of the structure of a Kafka-based concurrent consumption device provided in an embodiment of this application;

[0044] Figure 7 Exemplary systems that can be used to implement the various embodiments described in this application are shown. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] It should be understood that, in the description of this application, unless the context explicitly requires it, the words "comprising," "including," and similar terms throughout the specification and claims should be interpreted as encompassing rather than being exclusive or exhaustive; that is, meaning "including but not limited to."

[0047] It should be noted that the terms "S1," "S2," etc., are used only for descriptive purposes and do not specifically refer to the order or sequence, nor are they intended to limit this application. They are merely for the convenience of describing the method of this application and should not be construed as indicating the sequential order of the steps. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0048] Example 1

[0049] This application provides a concurrent consumption method based on Kafka, referring to... Figure 1 ,include:

[0050] S1. In response to the detection that the message retrieval processing thread has completed message retrieval, the message is grouped according to the topic and the topic's partition, and then sent to the message queue of the corresponding group of the topic.

[0051] S2. The message queue of each topic sends the corresponding message to the corresponding topic processing thread, and the topic processing thread submits the corresponding message to the partition thread pool for concurrent consumption.

[0052] S3. In response to the detection that all messages preceding any message to be committed have been consumed, the corresponding topic processing thread submits the offset of all messages preceding the current message to be committed to the commit queue, and the commit queue returns the message to the pull processing thread.

[0053] Specifically, refer to Figure 2 Previously, the Kafka client handled message retrieval (Poll), consumption (Process), and message acknowledgment (ACK) synchronously in a single thread, resulting in poor performance. The first step is to decouple message retrieval from consumption, specifically, decoupling the poll thread from the process thread. For example, refer to... Figure 3The `poll_processor` thread is responsible for handling poll and commit operations for one or more topics within the same cluster. First, the `poll_processor` pulls messages and polls them locally. Then, it groups the messages according to the topic and its partition, sending them to the corresponding topic queues (e.g., `pic1_message_queue`, `topic2_message_queue`, etc.). Each topic's message queue can directly submit messages to the partition thread pool for concurrent consumption, or the topic processing thread can submit the corresponding messages to the partition thread pool for concurrent consumption. Within the corresponding partition thread pool's `partition_poll`, a number of threads for each partition are started to sequentially consume messages from their respective partitions. Then, the offsets are batch-committed to the `commit_queue`, finally returning to the `poll_processor` thread for batch committing. When a topic's message queue directly submits messages to the partition thread pool for concurrent consumption, the poll thread and the process thread are already isolated, meaning they are decoupled, enabling asynchronous partition consumption and improving concurrency. Furthermore, sending messages to the corresponding topic processing thread via the topic's message queue, and then having the topic processing thread (topic_processor) submit the messages to the partition thread pool (partition_poll) for concurrent consumption, further improves concurrency, enhances consumption performance, and achieves multi-threaded concurrent consumption.

[0054] In some implementations, the method further includes:

[0055] S4. In response to the detection that any message to be committed and all messages preceding it in the order of the message to be committed have been consumed, the corresponding topic processing thread submits the offset of the current message to be committed and all messages preceding it in the order of the current message to be committed to the commit queue, and the commit queue returns it to the pull processing thread.

[0056] For example, refer to Figure 4Messages 0, 1, 2, 4, and 8 have been processed, while messages 3, 5, 6, 7, and 9 are not yet processed. Due to Kafka's offset commit mechanism and the out-of-order message processing by the thread pool, some messages that are later in the queue have been consumed, while some messages in the middle have not. In this case, if the later consumed messages are committed, the other unconsumed messages will be lost. For example, if message 8 is committed, unconsumed messages 3, 5, 6, and 7 may be lost. If message3 is the current message to be committed, and it is currently unconsumed, then message3 cannot be committed. However, if all messages preceding the current message to be committed (message0, message1, and message2) are detected as consumed, then message0, message1, and message2 can be committed (i.e., excluding the current message to be committed). Similarly, if message2 is the current message to be committed, and it is currently consumed, and simultaneously, all messages preceding the current message to be committed (message0 and message1) are also consumed, then message0, message1, and message2 can be committed (i.e., including the current message to be committed). Additionally, if message4 is the currently pending message, since message3 has not yet been consumed, only messages0, 1, and 2 can be committed. In this case, you can either commit messages0, 1, and 2, or wait for message3 to finish being consumed and its status updated before committing messages0, 1, 2, 3, and 4. Specifically, the specific commit configuration can be configured in the batch commit parameter settings. You can configure it to commit only after all messages in the commit queue (e.g., messages0 to 9 above) have been consumed, or to commit only after the first N messages (N≥1) in sequence have been consumed. This allows for flexible configuration to meet business needs based on the user's specific scenario.

[0057] In some implementations, the method further includes:

[0058] 101. Generate a bitset of the corresponding number of messages to be submitted based on the batch submission parameter configuration. Each status bit in the bitset is used to describe the consumption status of the messages to be submitted in the corresponding order.

[0059] 102. In response to detecting a change in the consumption status of a message to be submitted, update the corresponding status bit of the message to be submitted.

[0060] Specifically, a sliding window can be used to control message submission. This is implemented by instantiating a bitset of length `max.poll.records` configured for the topic after consumption begins. This bitset is used to map the message offset to its submission status, i.e., its consumption status. It's important to note that the batch commit parameter configuration `batchcommit` specifies the number of messages to commit per batch (e.g., ...). Figure 4 The set contains 10 messages (messages 0-9), which can be configured to 10, 30, 50, or 100 messages based on actual business scenarios and performance requirements. When the consumption status of a message to be committed changes, the corresponding status bit in the bitset is updated to accurately describe its current consumption status. For example, if the current consumption status of a message to be committed is "unconsumed," the corresponding status bit in the bitset will describe it as "unconsumed." If the consumption status changes to "consumed," the corresponding status bit in the bitset will also be updated to describe the current consumption status as "consumed." Then, the topic processing thread (topic_processor) checks the consecutive status bits in the bitset to confirm whether the conditions for initiating a commit are met.

[0061] In some implementations, the status bit includes at least one of 0 and 1, wherein 0 is used to describe the consumption status as unconsumed and 1 is used to describe the consumption status as consumed.

[0062] In some implementations, based on this, S3 includes:

[0063] In response to the detection that the status bits in the bitset corresponding to all messages before any message to be committed are 1, the corresponding topic processing thread submits the offset corresponding to all messages before the current message to be committed to the commit queue, and the commit queue returns the information to the pull processing thread.

[0064] S4 includes:

[0065] In response to the detection that the status bits in the bitset corresponding to any message to be committed and all messages preceding the message to be committed are 1, the corresponding topic processing thread submits the offset corresponding to the current message to be committed and all messages preceding the message to be committed to the commit queue, and the commit queue returns it to the pull processing thread.

[0066] For example, refer to Figure 5 If message0, message1, message2, message4, and message8 have been consumed, then the status bit in the corresponding bitset is 1; if message3, message5, message6, message7, and message9 have not yet been consumed, then the status bit in the corresponding bitset is 0. If message3 is the current message to be committed, and its current status bit is 0, then message3 is unconsumed and cannot be committed. However, if the status bits of all messages preceding the current message to be committed (message0, message1, and message2) are 1, meaning they are all consumed, then messages0, message1, and message2 can be committed. If message3's consumption status is updated to consumed, the status bit in the bitset is changed to "1111100010". Then, the topic processing thread (topic_processor) checks the consecutive status bits in the bitset and finds that messages0, message1, message2, message3, and message4 can all be committed. At this point, the batchcommit parameter can be configured to decide whether to commit. Generally, taking the case where all messages in the commit queue for the current offset have been consumed (i.e., all status bits in the corresponding bitset are 1, meaning the bitset is "1111111111") as an example, the offset of this batch of messages is then committed to the commit queue. The bitset status bits are then initialized, for example, changed to "0000000000". Then, based on the consumption status of a new batch of messages to be committed, the status bits are synchronously changed until the commit condition is met. It should be noted that the above explanation is only an example; in practice, the batch commit parameters can be flexibly configured to ensure that the first N (N≥1) status bits in the bitset are all 1 before committing.

[0067] In some implementations, the method further includes:

[0068] In response to the detection that the current time in the commit queue is more than a preset time threshold from the previous received offset time, the status information of the bitset is obtained;

[0069] Logs are generated based on the status information of the bitset, and the logs are returned to the pull processing thread.

[0070] Specifically, the batch commit parameter configuration includes at least one of the following: the number of messages committed per batch, the number of bits in the bitset, and a preset time threshold. The batch commit parameter configuration can simultaneously configure the number of bits in the bitset corresponding to the number of messages committed per batch, the batch commit time interval, the preset time threshold, and other parameters. When the topic processing thread `topic_processor` detects that the current time in the commit queue is more than the preset time threshold from the previous received offset, it indicates that one or more bits in the batch of messages to be committed may have a problem, requiring the generation of error information. At this time, the bitset status information is obtained, and a log is generated based on the bitset status information. This log describes which messages to be committed in the current batch (i.e., those whose corresponding bitset status is still 0) may have problems, identifies these messages to be committed, and provides error information such as the commit wait time. This log is then returned to the pull processing thread `poll_processor` to send an alert to the user, reminding the user or relevant operators to perform maintenance.

[0071] In this embodiment, the poll thread and the process thread are isolated and decoupled, enabling asynchronous partition consumption and improving concurrency. Simultaneously, messages are sent to the corresponding topic processing thread via the topic's message queue. The topic processing thread (topic_processor) then submits the messages to the partition thread pool (partition_poll) for concurrent consumption. Offset submission is only performed after all messages preceding the current message to be submitted have been consumed, thus preventing the loss of unconsumed messages. This further improves concurrency, enhances consumption performance, and enables multi-threaded concurrent consumption.

[0072] Example 2

[0073] Corresponding to the above embodiments, this application also provides a concurrent consumption device based on Kafka, referring to... Figure 6 The device includes: a message retrieval and distribution module, a message processing module, and an offset submission module.

[0074] The message retrieval and distribution module is configured to, in response to detecting that the retrieval processing thread has completed message retrieval, send the message to the message queue of the corresponding group of the topic according to the topic and the partition group of the topic; the message processing module is configured to send the corresponding message to the corresponding topic processing thread for each topic's message queue, and the topic processing thread submits the corresponding message to the partition thread pool for concurrent consumption; the offset commit module is configured to, in response to detecting that the consumption status of all messages before any message to be committed is consumed, submit the offset corresponding to all messages before the current message to be committed to the commit queue to the commit queue, and the commit queue returns the message to the retrieval processing thread.

[0075] Furthermore, the offset submission module is also used to respond to the detection that the consumption status of any message to be submitted and all messages preceding the message to be submitted are consumed, and the corresponding topic processing thread submits the offset of the current message to be submitted and all messages preceding the message to be submitted to the commit queue, and the commit queue returns it to the pull processing thread.

[0076] Furthermore, the offset submission module is also configured to generate a bitset of the corresponding number of messages to be submitted according to the batch submission parameters, wherein each status bit in the bitset is used to describe the consumption status of the messages to be submitted in the corresponding order; and to update the corresponding status bit of the messages to be submitted in response to detecting a change in the consumption status of the messages to be submitted.

[0077] Furthermore, the status bit includes at least one of 0 and 1, wherein 0 is used to describe the consumption status as unconsumed and 1 is used to describe the consumption status as consumed.

[0078] Furthermore, the offset submission module is also configured to, in response to detecting that the status bits in the bitset corresponding to all messages preceding the current message to be submitted are 1, submit the offsets corresponding to all messages preceding the current message to be submitted to the commit queue, and have the commit queue return to the pull processing thread; and to, in response to detecting that the status bits in the bitset corresponding to all messages preceding the current message to be submitted are 1, submit the offsets corresponding to the current message to be submitted to the commit queue, and have the commit queue return to the pull processing thread.

[0079] Furthermore, the offset submission module is also used to obtain the status information of the bitset in response to detecting that the current time of the commit queue is more than a preset time threshold away from the previous offset receiving time; and to generate a log based on the status information of the bitset, and return the log to the pull processing thread.

[0080] Furthermore, the batch submission parameter configuration includes at least one of the following: the number of messages submitted per batch, the number of bits in the bitset, and a preset time threshold.

[0081] Specific limitations regarding the Kafka-based concurrent consumption device can be found in the relevant limitations of the Kafka-based concurrent consumption method embodiments described above, and therefore are not limited here. Each module in the aforementioned Kafka-based concurrent consumption device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0082] Example 3

[0083] Corresponding to the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can implement the above-described concurrent consumption method based on Kafka.

[0084] like Figure 7As shown, in some embodiments, the system can serve as any of the aforementioned electronic devices for a Kafka-based concurrent consumption method in each of the described embodiments. In some embodiments, the system may include one or more computer-readable media (e.g., system memory or NVM / storage device) having instructions and one or more processors (e.g., processor(s)) coupled to the one or more computer-readable media and configured to execute the instructions to implement the module and thereby perform the actions described in this application.

[0085] In one embodiment, the system control module may include any suitable interface controller to provide any suitable interface to at least one of the processors(s) and / or any suitable device or component communicating with the system control module.

[0086] The system control module may include a memory controller module to provide an interface to the system memory. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0087] System memory can be used, for example, to load and store data and / or instructions for the system. In one embodiment, system memory may include any suitable volatile memory, such as suitable DRAM. In some embodiments, system memory may include Double Data Rate Type Quad Synchronous Dynamic Random Access Memory (DDR4 SDRAM).

[0088] In one embodiment, the system control module may include one or more input / output (I / O) controllers to provide interfaces to the NVM / storage device and (one or more) communication interfaces.

[0089] For example, an NVM / storage device can be used to store data and / or instructions. An NVM / storage device may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0090] NVM / storage devices may include storage resources that are physically part of a device on which the system is mounted, or that can be accessed by the device without necessarily being part of it. For example, an NVM / storage device may be accessed over a network via one or more communication interfaces.

[0091] One or more communication interfaces may provide the system with an interface to communicate over one or more networks and / or with any other suitable device. The system may wirelessly communicate with one or more components of a wireless network in accordance with any of the standards and / or protocols in one or more wireless network standards and / or protocols.

[0092] In one embodiment, at least one of the processors may be logically packaged with one or more controllers of the system control module (e.g., a memory controller module). In one embodiment, at least one of the processors may be logically packaged with one or more controllers of the system control module to form a system-in-package (SiP). In one embodiment, at least one of the processors may be integrated with the logic of one or more controllers of the system control module on the same die. In one embodiment, at least one of the processors may be integrated with the logic of one or more controllers of the system control module on the same die to form a system-on-a-chip (SoC).

[0093] In various embodiments, the system may be, but is not limited to, a server, workstation, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, the system may have more or fewer components and / or different architectures. For example, in some embodiments, the system includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0094] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0095] Furthermore, a portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0096] Communication media include media through which communication signals containing, for example, computer-readable instructions, data structures, program modules, or other data are transmitted from one system to another. Communication media can include guided transmission media (such as cables and wires (e.g., optical fibers, coaxial cables, etc.)) and wireless (unguided transmission) media capable of propagating energy waves, such as sound, electromagnetic, RF, microwave, and infrared. Computer-readable instructions, data structures, program modules, or other data can be embodied as modulated data signals in, for example, wireless media (such as carrier waves or similar mechanisms embodied as part of spread spectrum technology). The term "modulated data signal" refers to a signal whose one or more characteristics are altered or set in a manner that encodes information in the signal. Modulation can be analog, digital, or a hybrid modulation technique.

[0097] Herein, one embodiment of this application includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to run a method and / or technical solution based on the foregoing embodiments of this application.

[0098] Example 4

[0099] Corresponding to the above embodiments, this application also provides a computer-readable storage medium storing computer-executable instructions for executing a Kafka-based concurrent consumption method.

[0100] In this embodiment, a computer-readable storage medium may include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. For example, a computer-readable storage medium includes, but is not limited to, volatile memories such as random access memory (RAM, DRAM, SRAM); and non-volatile memories such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other currently known media or those developed hereafter capable of storing computer-readable information / data for use by a computer system.

[0101] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0102] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A Kafka-based concurrent consumption method, characterized in that, include: In response to the detection that the message retrieval processing thread has completed message retrieval, the message is sent to the message queue of the topic corresponding to the topic group according to the topic and the partition group of the topic; The message queue of each topic sends the corresponding message to the corresponding topic processing thread, and the topic processing thread submits the corresponding message to the partition thread pool corresponding to the topic for concurrent consumption, with one partition thread pool consuming multiple partitions under the topic; In response to the detection that all messages preceding any message in the sliding window to be submitted have been consumed, the corresponding topic processing thread submits the offset of all messages preceding the current message to be submitted to the commit queue, and the commit queue returns them to the pull processing thread, which then submits them in batches. The method further includes: A bitset of bits corresponding to the number of messages to be submitted is generated according to the batch submission parameter configuration. Each status bit in the bitset is used to describe the consumption status of the messages to be submitted in the corresponding order. In response to detecting a change in the consumption status of the message to be submitted, the status bit corresponding to the message to be submitted is updated.

2. The Kafka-based concurrent consumption method of claim 1, wherein, The method further includes: In response to the detection that any message to be committed and all messages preceding it in the order of the message to be committed have been consumed, the corresponding topic processing thread submits the offset of the current message to be committed and all messages preceding it in the order of the current message to be committed to the commit queue, and the commit queue returns it to the pull processing thread.

3. The Kafka-based concurrent consumption method of claim 1, wherein, The status bit includes at least one of 0 and 1, where 0 is used to describe the consumption status as unconsumed and 1 is used to describe the consumption status as consumed.

4. The Kafka-based concurrent consumption method of claim 3, wherein, The response to detecting that all messages preceding any message in the commit order have been consumed, the corresponding topic processing thread commits the offsets of all messages preceding the current commit order to the commit queue, and the commit queue returns the information to the pull processing thread, including: In response to the detection that all messages preceding any message to be committed have a status bit of 1 in the bitset, the topic processing thread submits the offset of all messages preceding the current message to be committed to the commit queue, and the commit queue returns the information to the pull processing thread. In response to detecting that the consumption status of any message to be committed and all messages preceding the message to be committed are consumed, the corresponding topic processing thread submits the offset of the currently pending message and all messages preceding the message to be committed to the commit queue, and the commit queue returns the information to the pull processing thread, including: In response to the detection that the status bits in the bitset corresponding to any message to be committed and all messages preceding the message to be committed are 1, the topic processing thread submits the offset corresponding to the current message to be committed and all messages preceding the message to be committed to the commit queue, and the commit queue returns it to the pull processing thread.

5. The method for Kafka-based concurrent consumption according to claim 1, wherein, The method further includes: In response to detecting that the current time of the commit queue is more than a preset time threshold from the previous received offset time, the status bit information of the bitset is obtained; Logs are generated based on the status information of the bitset, and the logs are returned to the fetch processing thread.

6. The Kafka-based concurrent consumption method of claim 5, wherein, The batch submission parameter configuration includes at least one of the following: the number of messages submitted per batch, the number of bits in the bitset, and a preset time threshold.

7. A Kafka-based concurrent consumption apparatus for implementing the Kafka-based concurrent consumption method of any one of claims 1-6. The device includes: The message retrieval and distribution module is used to respond to the detection that the retrieval processing thread has completed message retrieval, and to send the message to the message queue of the topic in the corresponding group according to the topic and the partition group of the topic; The message processing module is used to send the corresponding message from the message queue of each topic to the corresponding topic processing thread, and the topic processing thread submits the corresponding message to the partition thread pool for concurrent consumption. The offset submission module is used to respond to the detection that all messages preceding any message to be submitted in the order of messages to be submitted have been consumed. The corresponding topic processing thread submits the offset of all messages preceding the current message to be submitted to the commit queue, and the commit queue returns the message to the pull processing thread.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the Kafka-based concurrent consumption method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing computer-executable instructions, the computer-executable instructions comprising: The computer-executable instructions are used to execute the Kafka-based concurrent consumption method according to any one of claims 1 to 6.

Images