Kafka data landing method, device, equipment, storage medium and program product

Abstract

The disclosure provides a Kafka data landing method, relates to the technical field of distribution, and can be applied to the technical field of finance. The method comprises the following steps: in response to a data landing operation of a Kafka logical partition, intercepting a to-be-landed message; adding index information to the to-be-landed message, wherein the index information is used for locating the to-be-landed message; when it is determined that a preset packaging condition is met, putting to-be-landed messages of the same topic partition into a log block for packaging; determining a landing node of the log block according to a current disk remaining capacity; and routing the packaged log block to the landing node. The disclosure further provides a Kafka data landing device, equipment, a storage medium and a program product.

Description

Technical Field

[0001] This disclosure relates to the field of distributed technology, specifically to the field of data storage technology, and more specifically to a method, apparatus, device, storage medium, and program product for writing Kafka data to disk. Background Technology

[0002] Once a message enters Kafka, Kafka distributes it to a partition according to the specified partitioning strategy. In Kafka, the Broker process is responsible for moving and storing data. When the Broker process receives data, it stores it in a separate local directory. Because it's a local directory, uneven distribution of topics in the Kafka cluster or significant differences in topic traffic can lead to data skew.

[0003] In related technologies, when data skew is detected, it is impossible to quickly avoid the problem through emergency scaling. Manual intervention is required to resolve the issue by adjusting the partitioning strategy in the code or adjusting the distribution of topics. Therefore, there is an urgent need for a self-balancing message balancing mechanism and method for disk persistence.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] In view of the above problems, this disclosure provides a method, apparatus, device, storage medium and program product for achieving self-balancing data persistence to Kafka disk.

[0006] According to a first aspect of this disclosure, a method for persisting Kafka data to disk is provided, the method comprising:

[0007] In response to data write-to-disk operations for Kafka logical partitions, intercept messages awaiting write-to-disk execution.

[0008] Add index information to the message to be written to disk, the index information being used to locate the message to be written to disk;

[0009] Once the preset packaging conditions are met, messages to be written to disk from the same topic partition are packaged into log blocks.

[0010] The log block's write node is determined based on the current remaining disk capacity; and

[0011] The packaged log blocks are routed to the disk node.

[0012] According to embodiments of this disclosure, the preset packaging conditions include the data volume of the log block containing the message to be written to disk being greater than a first preset threshold or the waiting time for the message to be written to disk being greater than a second preset threshold.

[0013] According to embodiments of this disclosure, adding index information to the message to be written to disk includes:

[0014] The message to be written to disk is parsed to obtain the target topic and target partition of the message to be written to disk;

[0015] Calculate the hash value based on the target topic and the target partition; and

[0016] The hash value is inserted as index information into the header of the message body of the message to be written to disk.

[0017] According to embodiments of this disclosure, determining the disk write node for the log block based on the current remaining disk capacity includes:

[0018] Get the remaining disk capacity of the current node;

[0019] The number of replicas for the first target nodes is determined based on the remaining disk capacity of the current node, wherein the number of replicas is the number of copies of Kafka data that need to be synchronized; and

[0020] The node with the largest remaining disk capacity among the first target nodes is selected as the node where the log block is written to disk.

[0021] According to embodiments of this disclosure, before routing the packaged log blocks to the disk persistence node, the method further includes:

[0022] Based on the aforementioned replicas, several first target nodes add push tags to the already packaged log blocks.

[0023] According to embodiments of this disclosure, it further includes:

[0024] The log blocks are synchronized according to the push label.

[0025] A second aspect of this disclosure provides a Kafka data persistence device, the device comprising:

[0026] The message interception module is used to intercept messages waiting to be written to disk in response to data write operations of Kafka logical partitions;

[0027] The message processing module is used to add index information to the message to be written to disk, and the index information is used to locate the message to be written to disk;

[0028] The message packaging module is used to package messages to be written to disk in the same topic partition into log blocks after it is determined that the preset packaging conditions are met.

[0029] The disk write node determination module is used to determine the disk write node of the log block based on the current remaining disk capacity; and

[0030] The message routing module is used to route the packaged log blocks to the disk node.

[0031] According to embodiments of this disclosure, the preset packaging conditions include the data volume of the log block containing the message to be written to disk being greater than a first preset threshold or the waiting time for the message to be written to disk being greater than a second preset threshold.

[0032] According to embodiments of this disclosure, the message processing module includes: a parsing submodule, a calculation submodule, and an index insertion submodule.

[0033] The parsing submodule is used to parse the message to be written to disk to obtain the target topic and target partition of the message to be written to disk;

[0034] A calculation submodule is used to calculate a hash value based on the target topic and the target partition; and

[0035] The index insertion submodule is used to insert the hash value as index information into the message header of the message to be written to disk.

[0036] According to embodiments of this disclosure, the disk placement node determination module includes: an acquisition submodule, a first determination submodule, and a second determination submodule.

[0037] The `get` submodule is used to obtain the remaining disk capacity of the current node;

[0038] The first determining submodule is used to determine the number of replicas for the first target nodes based on the remaining disk capacity of the current node, wherein the number of replicas is the number of copies of Kafka data that need to be synchronized; and

[0039] The second determining submodule is used to select the node with the largest remaining disk capacity among the first target nodes as the disk placement node for the log block.

[0040] According to embodiments of this disclosure, the apparatus further includes a push tag adding module.

[0041] The push tag adding module is used to add push tags to the packaged log blocks according to the replica number of first target nodes;

[0042] According to embodiments of this disclosure, the apparatus further includes a data synchronization module.

[0043] The data synchronization module is used to synchronize the log block according to the push tag.

[0044] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the Kafka data persistence method described above.

[0045] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described Kafka data persistence method.

[0046] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described Kafka data persistence method.

[0047] This disclosure provides a method for persisting Kafka data to disk. Before persistence, data is intercepted, and idle nodes are selected based on the remaining disk capacity. After meeting preset packaging conditions, messages from the same topic partition are packaged into log blocks and routed to the persistence nodes. Index information is added to the messages before routing to facilitate message consumption. Compared to related technologies, this method achieves self-balancing of Kafka data storage without human intervention, resolving data storage skew caused by uneven topic distribution or varying message traffic. Attached Figure Description

[0048] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0049] Figure 1a This diagram illustrates a Kafka data storage structure in related technologies.

[0050] Figure 1b This diagram illustrates the original Kafka message flow process.

[0051] Figure 2a The illustration schematically depicts an application scenario of a Kafka data persistence method, apparatus, device, storage medium, and program product according to embodiments of this disclosure;

[0052] Figure 2b This diagram schematically illustrates the system architecture of a Kafka data persistence device provided according to an embodiment of the present disclosure.

[0053] Figure 3 A flowchart illustrating a method for persisting Kafka data to disk according to an embodiment of this disclosure is shown schematically.

[0054] Figure 4 This illustration schematically shows a flowchart of adding index information to the message to be written to disk according to an embodiment of the present disclosure;

[0055] Figure 5 This schematically illustrates a flowchart of determining the disk write node of the log block based on the current remaining disk capacity, according to an embodiment of the present disclosure.

[0056] Figure 6 This schematically illustrates a structural block diagram of a Kafka data persistence device according to an embodiment of the present disclosure;

[0057] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a Kafka data persistence method according to an embodiment of the present disclosure. Detailed Implementation

[0058] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0059] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0060] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0061] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0062] First, the terms appearing in the embodiments of this disclosure will be explained:

[0063] A message engine, often referred to as message middleware, is software that enables loosely coupled and asynchronous message delivery between different systems, allowing for the transmission of precisely semantic messages. System A sends a message to the message engine system, and system B reads the message sent by system A from the message engine system.

[0064] Message Broker: The message engine server, handling message service processes and related data. One Broker represents one message engine server providing services to the outside world.

[0065] Message Topic: A logical unit on the message engine that stores different types of messages, used to distinguish specific business logic. When a producer produces a message, it specifies the topic to which the message belongs. When a consumer consumes a message, it specifies the topic and retrieves the messages produced by the producer within that topic.

[0066] Partition: An ordered and immutable sequence of messages. Each topic can have multiple partitions.

[0067] Producer: An application that publishes messages to a topic.

[0068] Consumer: An application that subscribes to messages from a topic.

[0069] Client: The message engine client includes message producers and consumers.

[0070] To better understand the technical solutions and technical issues of this disclosure, firstly through... Figure 1a and Figure 1b This section introduces the Kafka data storage structure and the original Kafka message flow process. Figure 1a This diagram illustrates a Kafka data storage structure in related technologies. Figure 1b The flowchart illustrating the original Kafka message flow is shown in the diagram.

[0071] like Figure 1a and Figure 1bAs shown, Kafka partitions store data using a message log format. Kafka log files are divided into multiple log segments, each of equal size. Each log segment includes an offset index file, a data file, and a timestamp index file, storing multiple messages. Each message is appended to the latest log segment. When a log segment is full, Kafka automatically splits it into a new segment and archives the old one. In Kafka, when a producer produces messages to the Kafka controller node, the controller node distributes the messages to a partition according to the specified partitioning strategy. The Broker process is responsible for moving and storing the data. When the Broker process receives data, it stores it in a separate local directory. Because it's a local directory, uneven distribution of topics in the Kafka cluster or significant differences in topic traffic can lead to data skew.

[0072] Based on the above technical problems, embodiments of this disclosure provide a method for writing Kafka data to disk, the method comprising: intercepting messages to be written to disk in response to a data write-to-disk operation of a Kafka logical partition;

[0073] Add index information to the message to be written to disk, the index information is used to locate the message to be written to disk; when it is determined that the preset packaging conditions are met, put the messages to be written to disk in the same topic partition into log blocks for packaging; determine the disk writing node of the log block according to the current remaining disk capacity; route the packaged log block to the disk writing node.

[0074] Figure 2a The illustration schematically depicts an application scenario of a Kafka data persistence method, apparatus, device, storage medium, and program product according to embodiments of the present disclosure. Figure 2b The diagram illustrates the system architecture of a Kafka data persistence device according to an embodiment of the present disclosure.

[0075] like Figure 2a As shown, application scenario 100 according to this embodiment may include a Kafka data persistence scenario. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0076] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0077] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0078] Server 105 can be a Kafka node server. This server has an embedded data processing device that intercepts and processes message data before it is written to disk. It determines the actual disk write node based on the remaining disk capacity, thereby achieving self-balancing of Kafka data write to disk.

[0079] It should be noted that the Kafka data persistence method provided in this embodiment can generally be executed by server 105. Correspondingly, the Kafka data persistence device provided in this embodiment can generally be located in server 105. The Kafka data persistence method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the Kafka data persistence device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0080] It should be understood that Figure 2a The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0081] It should be noted that the Kafka data persistence method and apparatus determined in the embodiments of this disclosure can be used in the field of distributed technology, the field of financial technology, and any field other than the financial field. The application field of the Kafka data persistence method and apparatus determined in the embodiments of this disclosure is not limited.

[0082] like Figure 2bAs shown in the embodiments of this disclosure, the Kafka data persistence device includes a message interception module, a message processing module, a message routing module, and a message persistence synchronization module. When a logical partition allocates received messages according to the original partitioning logic, the message interception module intercepts the persistence messages and sends them to the message processing module for processing. The message processing module sets an index based on the topic and partition of the message to be persisted to disk. The message routing module packages the messages that meet the packaging conditions and routes them to the node with the lowest current storage pressure. Finally, the message persistence synchronization module performs data synchronization.

[0083] The following will be based on Figure 2a The scene described and Figure 2b The system architecture described above, through Figures 3-5 The present disclosure provides a detailed description of the method for persisting Kafka data to disk according to embodiments.

[0084] Figure 3 A flowchart illustrating a method for persisting Kafka data to disk according to an embodiment of this disclosure is shown schematically. Figure 3 As shown, the Kafka data persistence method in this embodiment includes operations S210 to S250, which can be executed by a server or other computing devices.

[0085] When operating S210, in response to the data write-to-disk operation of the Kafka logical partition, intercept messages waiting to be written to disk.

[0086] In one example, to address the data skew issue during Kafka data write-to-disk, this embodiment modifies the original Kafka message distribution and write-to-disk logic (the original Kafka Controller node message distribution logic). Specifically, when the Kafka logical partition Controller node receives a new message, it no longer distributes the message according to the preset partitioning strategy. Instead, the message interceptor intercepts the message and processes it according to the current disk capacity to achieve self-balancing of data storage.

[0087] In operation S220, index information is added to the message to be written to disk.

[0088] According to embodiments of this disclosure, the index information is used to locate the message to be written to disk.

[0089] In one example, since messages are no longer stored according to the original disk write logic, to avoid affecting normal message consumption on the consumer side, it is necessary to add index information to each message to be written to disk. This index information can be used to locate the message to be written to disk. For details on adding the index, please refer to [link to relevant documentation]. Figure 4 Operations S221 to S223 shown will not be described again here.

[0090] When operating S230, once the preset packaging conditions are met, the messages to be written to disk in the same topic partition are placed into the log block for packaging.

[0091] According to an embodiment of this disclosure, the preset packaging conditions include the data volume of the log block containing the message to be written to disk being greater than a first preset threshold or the waiting time for the message to be written to disk being greater than a second preset threshold.

[0092] In one example, an existing Kafka partition contains multiple message logs, each divided into multiple log segments of equal size. Each log segment includes an offset index file, a data file, and a timestamp index file, storing multiple messages. Each message is appended to the latest log segment. When a log segment is full, Kafka automatically splits it into a new segment and archives the old one. In this embodiment, partitions are no longer used as the physical storage unit; that is, each partition is no longer stored as a single entity on a single broker. Instead, each partition is more finely granular, using log segments as the physical storage unit. Each log segment stores one or more messages. Each log segment is received, processed, packaged, and routed to a designated node via Kafka's data persistence mechanism. This ensures that a logically ordered set of partitioned data is physically evenly distributed across multiple storage nodes. It should be noted that the "log block" referred to in this disclosure is the same as the "log segment" in Kafka technology in terms of physical file structure, but the disk node for writing the log block is no longer fixed, but is dynamically selected based on the current remaining disk capacity; in order to distinguish it from related technologies, this disclosure refers to it as "log block".

[0093] To maintain partition order, messages from the same topic partition are placed in the same log segment. When preset packaging conditions are met, the log segment containing one or more messages awaiting disk writes is packaged. These preset packaging conditions include: the data volume of the log segment containing the messages awaiting disk writes exceeding a first preset threshold (i.e., the log segment reaches its storage limit); or the waiting time for the messages awaiting disk writes exceeding a second preset threshold. In some business scenarios, message production rates may be very low, causing log segments to remain at their storage limit for an extended period. To avoid impacting normal consumer consumption, log segments are packaged when the waiting time for the messages awaiting disk writes exceeds the second preset threshold. The first and second preset thresholds can be customized according to user needs. If message timeliness is critical, the second preset threshold can be set lower to achieve immediate delivery. If timeliness is less critical, the second preset threshold can be set higher. If the preset packaging conditions are not met, the data is temporarily stored, waiting for new data until the preset packaging conditions are met.

[0094] In operation S240, the disk write node for the log block is determined based on the current remaining disk capacity.

[0095] In operation S250, the packaged log blocks are routed to the disk-writing node.

[0096] In one example, the log block write node is determined based on the current remaining disk capacity. The node with the largest remaining disk capacity is prioritized as the write node, and the packaged log blocks are routed to that write node. For details, please refer to [link to documentation]. Figure 5 Operations S241 to S243 are shown.

[0097] This disclosure provides a method for persisting Kafka data to disk. Before persistence, data is intercepted, and idle nodes are selected based on the remaining disk capacity. After meeting preset packaging conditions, messages from the same topic partition are packaged into log blocks and routed to the persistence nodes. Index information is added to the messages before routing to facilitate message consumption. Compared to related technologies, this method achieves self-balancing of Kafka data storage without human intervention, resolving data storage skew caused by uneven topic distribution or varying message traffic.

[0098] The following will be through Figure 4 This invention describes the process of adding index information to messages to be written to disk in the embodiments of this disclosure. Figure 4 The illustration shows a flowchart of adding index information to the message to be written to disk according to an embodiment of this disclosure. For example... Figure 4As shown, operation S220 includes operations S221 to S223.

[0099] In operation S221, the message to be written to disk is parsed to obtain the target topic and target partition of the message to be written to disk.

[0100] In operation S222, a hash value is calculated based on the target topic and the target partition.

[0101] In operation S223, the hash value is inserted as index information into the message header of the message to be written to disk.

[0102] In one example, when the message processing module receives a message to be written to disk, it first parses the received message, parsing the corresponding Topic and Partition to be sent, thus determining the target topic and target partition of the message to be written to disk. Then, based on these two values, it obtains the number of corresponding Segments to be packaged, specifically calculating the hash value based on the target topic and target partition, and then inserting Topic-Partition-Segment into the message body header as an index.

[0103] Figure 5 The flowchart illustrating the process of determining the log block's write node based on the current remaining disk capacity, according to an embodiment of this disclosure, is shown in the illustration.

[0104] like Figure 5 As shown, operation S240 includes operations S241 to S243.

[0105] In operation S241, obtain the remaining disk capacity of the current node.

[0106] In operation S242, the number of replica first target nodes is determined based on the remaining disk capacity of the current node.

[0107] According to embodiments of this disclosure, the number of replicas is the number of copies of Kafka data that need to be synchronized.

[0108] In operation S243, the node with the largest remaining disk capacity among the first target nodes is selected as the node for writing the log block to disk.

[0109] In one example, by monitoring disk capacity data in real time, the remaining disk capacity of all nodes is obtained. Based on the remaining disk capacity of the current node, multiple first target nodes are determined. The number of first target nodes is the number of replicas, that is, the number of copies of Kafka data that need to be synchronized. For example, if the number of replicas is 3, then the 3 first target nodes with the largest remaining disk capacity are determined based on the remaining disk capacity of the current node, and the node with the largest remaining disk capacity among the first target nodes is used as the disk write node.

[0110] According to embodiments of this disclosure, push tags are added to the packaged log blocks by the replica plurality of first target nodes. The log blocks are then synchronized based on the push tags.

[0111] To ensure high availability of Kafka, after log blocks are stored on the disk-writing nodes, the data needs to be synchronously backed up according to a preset number of replicas. Specifically, after identifying the nodes with the least storage pressure, the node information is added as a push tag to the packaged log block. The log block is then pushed to the message disk-writing synchronization module corresponding to the disk-writing node. This module parses the push tag on the segment, and after confirming that it is correct, it starts calling the original Kafka disk-writing operation to write the data to disk. At the same time, it pushes the data to other nodes to write the data to disk according to the segment push tag.

[0112] Based on the above-described method for persisting Kafka data to disk, this disclosure also provides a device for persisting Kafka data to disk. The following will combine... Figure 6 The device is described in detail.

[0113] Figure 6 The diagram illustrates a structural block diagram of a Kafka data persistence device according to an embodiment of the present disclosure.

[0114] like Figure 6 As shown, the Kafka data persistence device 700 in this embodiment includes a message interception module 710, a message processing module 720, a message packaging module 730, a persistence node determination module 740, and a message routing module 750.

[0115] The message interception module 710 is used to intercept messages to be written to disk in response to data write operations of Kafka logical partitions. In one embodiment, the message interception module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0116] The message processing module 720 is used to add index information to the message to be written to disk, and the index information is used to locate the message to be written to disk. In one embodiment, the message processing module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0117] The message packaging module 730 is used to package messages to be written to disk from the same topic partition into log blocks after determining that the preset packaging conditions are met. In one embodiment, the message packaging module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0118] The disk persistence node determination module 740 is used to determine the disk persistence node of the log block based on the current remaining disk capacity. In one embodiment, the disk persistence node determination module 740 can be used to perform the operation S240 described above, which will not be repeated here.

[0119] The message routing module 750 is used to route the packaged log blocks to the disk persistence node. In one embodiment, the message routing module 750 can be used to perform the operation S250 described above, which will not be repeated here.

[0120] According to embodiments of this disclosure, the message processing module includes: a parsing submodule, a calculation submodule, and an index insertion submodule.

[0121] The parsing submodule is used to parse the message to be written to disk to obtain the target topic and target partition of the message. In one embodiment, the parsing submodule can be used to perform the operation S221 described above, which will not be repeated here.

[0122] The calculation submodule is used to calculate a hash value based on the target topic and the target partition. In one embodiment, the calculation submodule can be used to perform the operation S222 described above, which will not be repeated here.

[0123] The index insertion submodule is used to insert the hash value as index information into the message body header of the message to be written to disk. In one embodiment, the index insertion submodule can be used to perform the operation S223 described above, which will not be repeated here.

[0124] According to an embodiment of this disclosure, the disk placement node determination module 240 includes: an acquisition submodule, a first determination submodule, and a second determination submodule.

[0125] The acquisition submodule is used to obtain the remaining disk capacity of the current node. In one embodiment, the acquisition submodule can be used to perform the operation S241 described above, which will not be repeated here.

[0126] The first determining submodule is used to determine the number of first target nodes to be replicated based on the remaining disk capacity of the current node, wherein the number of replicas is the number of copies of Kafka data that need to be synchronized. In one embodiment, the first determining submodule can be used to execute the operation S242 described above, which will not be repeated here.

[0127] The second determining submodule is used to select the node with the largest remaining disk capacity among the first target nodes as the node for writing the log block to disk. In one embodiment, the second determining submodule can be used to perform the operation S243 described above, which will not be repeated here.

[0128] According to embodiments of this disclosure, the apparatus further includes a push tag adding module.

[0129] The push tag adding module is used to add push tags to the packaged log blocks based on the replica number of the first target nodes.

[0130] According to embodiments of this disclosure, the apparatus further includes a data synchronization module.

[0131] The data synchronization module is used to synchronize the log block according to the push tag.

[0132] According to embodiments of this disclosure, any multiple modules among the message interception module 710, message processing module 720, message packaging module 730, disk insertion node determination module 740, and message routing module 750 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the message interception module 710, message processing module 720, message packaging module 730, disk insertion node determination module 740, and message routing module 750 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the message interception module 710, message processing module 720, message packaging module 730, disk drop node determination module 740, and message routing module 750 can be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0133] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a Kafka data persistence method according to an embodiment of the present disclosure.

[0134] like Figure 7 As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0135] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0136] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0137] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The aforementioned computer-readable storage medium carries one or more programs, which, when executed, implement the Kafka data persistence method according to the embodiments of this disclosure.

[0138] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0139] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the Kafka data persistence method provided in the embodiments of this disclosure.

[0140] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0141] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0142] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0143] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0145] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0146] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for persisting Kafka data to disk, characterized in that, The method includes: In response to data write-to-disk operations for Kafka logical partitions, intercept messages awaiting write-to-disk execution. Add index information to the message to be written to disk, the index information being used to locate the message to be written to disk; Once the preset packaging conditions are met, messages to be written to disk from the same topic partition are packaged into log blocks. The log block's write node is determined based on the current remaining disk capacity; and The packaged log blocks are routed to the disk-writing node; The step of determining the log block's write node based on the current remaining disk capacity includes: Get the remaining disk capacity of the current node; The number of replicas for the first target nodes is determined based on the remaining disk capacity of the current node, wherein the number of replicas is the number of copies of Kafka data that need to be synchronized; and The node with the largest remaining disk capacity among the first target nodes is selected as the node where the log block is written to disk.

2. The Kafka data persistence method according to claim 1, characterized in that, The preset packaging conditions include the data volume of the log block containing the message to be written to disk being greater than a first preset threshold or the waiting time of the message to be written to disk being greater than a second preset threshold.

3. The Kafka data persistence method according to claim 2, characterized in that, Adding index information to the message to be written to disk includes: The message to be written to disk is parsed to obtain the target topic and target partition of the message to be written to disk; Calculate the hash value based on the target topic and the target partition; and The hash value is inserted as index information into the header of the message body of the message to be written to disk.

4. The Kafka data persistence method according to claim 1, characterized in that, Before routing the packaged log blocks to the disk-writing node, the process also includes: Based on the aforementioned replicas, several first target nodes add push tags to the already packaged log blocks.

5. The Kafka data persistence method according to claim 4, characterized in that, Also includes: The log blocks are synchronized according to the push label.

6. A Kafka data persistence device, characterized in that, The device includes: The message interception module is used to intercept messages waiting to be written to disk in response to data write operations of Kafka logical partitions; The message processing module is used to add index information to the message to be written to disk, and the index information is used to locate the message to be written to disk; The message packaging module is used to package messages to be written to disk in the same topic partition into log blocks after it is determined that the preset packaging conditions are met. The disk write node determination module is used to determine the disk write node of the log block based on the current remaining disk capacity; and The message routing module is used to route the packaged log blocks to the disk persistence node; The disk persistence node determination module is also used to obtain the remaining disk capacity of the current node; determine the number of replicas of the first target nodes based on the remaining disk capacity of the current node, wherein the number of replicas is the number of copies of Kafka data that need to be synchronized; and select the node with the largest remaining disk capacity among the first target nodes as the disk persistence node for the log block.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors execute the Kafka data persistence method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the Kafka data persistence method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program that, when executed by a processor, implements the Kafka data persistence method according to any one of claims 1 to 5.

Images