High-availability data exchange method and device based on multiple Flume nodes

Abstract

The invention relates to the technical field of distributed data acquisition and transmission, and particularly provides a high-availability data exchange method and device based on multiple Flume nodes, and the method comprises the steps: firstly, storing data to be exchanged as a starting point of data exchange, and receiving the exchanged data as an exchange terminal point of the data; at least two Flume nodes form a consumer group and jointly consume the same Topic of a source Kafka, a priority is allocated to each Flume node, a main node is defaulted to execute a data exchange task, when the main node fails, the main node is automatically switched to a low-priority standby node, and the main node takes over the task again after recovery. Compared with the prior art, the method has the advantages that high availability of the Flume process can be realized, task state abnormity caused by node shutdown is avoided, and stability and reliability of a data exchange system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed data collection and transmission, and specifically provides a high-availability data exchange method and device based on multiple Flume nodes. Background Art

[0002] In a data exchange scenario, Flume is often used to collect data from a message queue such as Kafka and transmit it to a target system. Existing technologies usually use a single Flume node for data exchange. When this node fails and shuts down, the status of the submitted tasks cannot be updated, resulting in the tasks being displayed as "running" for a long time, and the status synchronization can only be completed after the node recovers. This single-point failure problem seriously affects the system reliability and operation and maintenance efficiency.

[0003] In addition, existing redundancy solutions require manual switching to a standby node, and the mean time to repair (MTTR) is as long as 30 minutes, which is difficult to meet the high-availability requirements.

[0004] Although some solutions attempt to improve availability through redundant deployment, they lack an automated failover mechanism and are complex to configure. Therefore, there is an urgent need for a solution that can achieve high availability, automated failover, and simple configuration for Flume. Summary of the Invention

[0005] The present invention aims at the above-mentioned deficiencies of the prior art and provides a highly practical high-availability data exchange method based on multiple Flume nodes.

[0006] A further technical task of the present invention is to provide a highly available data exchange device based on multiple Flume nodes that is reasonably designed, safe, and applicable.

[0007] The technical solution adopted by the present invention to solve its technical problems is as follows:

[0008] For the high-availability data exchange method based on multiple Flume nodes, first, store the data to be exchanged as the starting point of data exchange, and receive the data after the exchange is completed as the end point of data exchange;

[0009] At least two Flume nodes form a consumer group, jointly consume the same Topic from the source Kafka, assign priorities to each Flume node, and by default, the primary node executes the data exchange task. When the primary node fails, it automatically switches to a standby node with a lower priority. After the primary node recovers, it takes over the task again.

[0010] Further, the Flume nodes are divided into Flume-1, Flume-2, and Flume-3. Flume-1, as a data collection node, reads data from the source Kafka cluster and sends the read data to Flume-2 and Flume-3 via the Avro protocol.

[0011] Further, in Flume-1, configure the Sink group g1, which includes Flume-2 with a priority of 0 and Flume-3 with a priority of 1; set the processor type to failover and specify the failover timeout parameter.

[0012] By default, Flume-2 acts as the primary node, responsible for writing data to the target Kafka cluster, and Flume-3 acts as the standby node, in a standby state, monitoring the running status of Flume-2 in real time.

[0013] Further, when Flume-2 fails or becomes unavailable due to a shutdown, the failover module in Flume-1 detects the anomaly through the heartbeat detection mechanism and automatically switches the data exchange task to Flume-3. After Flume-3 takes over the task, it continues to write data to the target Kafka cluster to ensure that the data exchange process is not interrupted.

[0014] When Flume-2 resumes operation, the failover module in Flume-1 detects that the primary node is available and automatically switches the task back to Flume-2. After Flume-2 takes over the task again, it updates the task status to "success" to ensure the real-time and accuracy of the task status.

[0015] Further, for the Flume-1 configuration, define the Sink group g1, which includes Flume-2 and Flume-3, set the processor type to failover, and specify the priority parameters. The priority of Flume-2 is 0, and the priority of Flume-3 is 1. Configure the failover timeout parameter to ensure that the task can be switched to the standby node in case of a primary node failure.

[0016] For the Flume-2 configuration, configure the Avro Sink, point to the target Kafka cluster, set a high priority, and execute the data exchange task as the primary node.

[0017] For the Flume-3 configuration, configure the Avro Sink, point to the target Kafka cluster, set a low priority, and wait as the standby node.

[0018] Furthermore, the Sink group mechanism of Flume synchronizes the task status of the primary and standby nodes in real time. When the primary node recovers, the task status is automatically synchronized to "success". An integrated monitoring tool is set up to monitor the running status of Flume nodes in real time. When a node failure is detected, an alarm is triggered and the operation and maintenance personnel are notified to handle it in a timely manner.

[0019] Furthermore, set global memory parameters in the flume-env.sh configuration file or dynamically adjust them in the startup command for memory optimization;

[0020] When performing batch processing in Flume, configure the batch processing parameters of Flume and write them to the target Kafka cluster in batches.

[0021] A highly available data exchange device based on multiple Flume nodes, characterized by including: at least one memory and at least one processor;

[0022] The at least one memory is used to store machine-readable programs;

[0023] The at least one processor is used to call the machine-readable program and execute a highly available data exchange method based on multiple Flume nodes.

[0024] Compared with the prior art, the highly available data exchange method and device based on multiple Flume nodes of the present invention have the following prominent beneficial effects:

[0025] The present invention realizes redundant backup through multiple Flume node groups, avoids single point of failure, and the failover mechanism ensures continuous execution of tasks and real-time update of status. High-availability configuration is achieved based on the native functions of Flume (such as Failover Sink group), reducing the deployment complexity, and improving the data exchange efficiency through batch processing and memory optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0027] Attached Figure 1 is a schematic flowchart of a highly available data exchange method based on multiple Flume nodes. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] To enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0029] The following gives an optimal embodiment:

[0030] Embodiment 1:

[0031] As Figure 1 shown, a high-availability data exchange method based on multiple Flume nodes in this embodiment first stores the data to be exchanged as the starting point of data exchange, and receives the data after the exchange is completed as the end point of data exchange. At least two Flume nodes form a consumer group to jointly consume the same Topic from the source Kafka.

[0032] A priority is assigned to each Flume node, and by default, the high-priority node (master node) executes the data exchange task. When the master node fails, it automatically switches to the low-priority standby node, and the master node takes over the task again after recovery.

[0033] The Flume nodes are divided into Flume1, Flume2, and Flume3. Flume-1 is used as a data collection node to read data (Topic: flumeTest) from the source Kafka cluster and send the read data to Flume-2 and Flume-3 through the Avro protocol.

[0034] Configure a Sink group (g1) in Flume-1, which includes Flume-2 (priority 0) and Flume-3 (priority 1);

[0035] Set the processor type to failover and specify the failover timeout parameter (maxpenalty = 10000, indicating the longest takeover waiting time of the standby node after the master node fails).

[0036] By default, Flume-2 is used as the master node to write data to the target Kafka cluster;

[0037] Flume-3 is used as a standby node, in a standby state, and monitors the running state of Flume-2 in real time.

[0038] When Flume-2 fails or stops due to a fault, the failover module of Flume-1 detects the abnormality through a heartbeat detection mechanism (such as a periodic heartbeat packet) and automatically switches the data exchange task to Flume-3;

[0039] After Flume-3 takes over the task, it continues to write data to the target Kafka cluster to ensure that the data exchange process is not interrupted.

[0040] When Flume-2 resumes operation, the failover module of Flume-1 detects that the master node is available and automatically switches the task back to Flume-2;

[0041] After Flume-2 takes over the task again, it updates the task status to "Success" to ensure the real-time and accuracy of the task status.

[0042] In the Flume-1 configuration, define a Sink group (g1) that includes Flume-2 and Flume-3, set the processor type to failover, and specify the priority parameter (Flume-2 priority is 0, Flume-3 priority is 1); configure the failover timeout parameter (maxpenalty=10000) to ensure that the server can quickly switch to the standby node when the primary node fails.

[0043] In the Flume-2 configuration, configure the Avro Sink, point to the target Kafka cluster, set a high priority (priority = 10), and use it as the master node to perform data exchange tasks.

[0044] In the Flume-3 configuration, configure the Avro Sink to point to the target Kafka cluster, set a low priority (priority = 1), and stand by as a backup node.

[0045] When performing status synchronization, the task status of the active and standby nodes is synchronized in real time through Flume's Sink group mechanism; when the active node is restored, the task status is automatically synchronized to "successful" to prevent the task from being displayed as "running" for a long time.

[0046] When performing monitoring and alerting, set up integrated monitoring tools (such as Prometheus) to monitor the running status of Flume nodes in real time;

[0047] When a node failure is detected, an alarm is triggered and the operation and maintenance personnel are notified to handle it in time.

[0048] Optimize node performance and ensure stability in high-concurrency scenarios by adjusting Flume memory parameters (such as -Xmx4g). Set global memory parameters in the flume-env.sh configuration file or adjust them dynamically in the startup command.

[0049] When performing Flume batch processing, configure the Flume batch processing parameters (such as batchSize = 1000, batchDurationMillis = 2000) to improve the data exchange efficiency. By writing to the target Kafka cluster in batches, network overhead and system load are reduced.

[0050] Example 2:

[0051] Deploy Flume-1 on Node 21 and configure it to read data from the Kafka Topic (flumeTest);

[0052] Deploy Flume-2 and Flume-3 on Nodes 22 and 23 respectively, and configure them as Avro Sinks, pointing to the target Kafka cluster;

[0053] Define a Sink group (g1) in Flume-1, which includes Flume-2 (priority 10) and Flume-3 (priority 1), and set the processor type to failover.

[0054] When Flume-2 shuts down, Flume-3 automatically takes over the data exchange task;

[0055] After Flume-2 resumes, the task automatically switches back to Flume-2, and the task status is updated to "successful" in real time.

[0056] Example 3:

[0057] A highly available data exchange device based on multiple Flume nodes, characterized in that it includes: at least one memory and at least one processor;

[0058] The at least one memory is used to store machine-readable programs;

[0059] The at least one processor is used to call the machine-readable program to execute the highly available data exchange method based on multiple Flume nodes.

[0060] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0061] The memory can be used to store computer programs and / or modules. By running or executing the computer programs and / or modules stored in the memory and invoking the data stored in the memory, the processor can implement various functions of the electronic device. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function, etc.; the data storage area can store the data created according to the use of the terminal, etc. In addition, the memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, memory, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash memory cards, at least one magnetic disk storage period, flash memory devices, or other volatile solid-state storage devices.

[0062] The above specific embodiments are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above specific embodiments. Any technical solution that conforms to the above specific embodiments of the present invention and any appropriate changes or substitutions made by those of ordinary skill in the art shall fall within the patent protection scope of the present invention.

[0063] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A high-availability data exchange method based on multiple Flume nodes, characterized in that First, store the data to be exchanged as the starting point of data exchange, and receive the data after the exchange is completed as the end point of data exchange; At least two Flume nodes form a consumer group, jointly consuming the same Topic from the source Kafka. Assign priorities to each Flume node. By default, the master node executes the data exchange task. When the master node fails, it automatically switches to a standby node with a lower priority. After the master node recovers, it takes over the task again.

2. The high-availability data exchange method based on multiple Flume nodes according to claim 1, wherein F The Flume nodes are divided into Flume-1, Flume-2, and Flume-3. Flume-1 is used as a data collection node, reads data from the source Kafka cluster, and sends the read data to Flume-2 and Flume-3 via the Avro protocol.

3. The high-availability data exchange method based on multiple Flume nodes according to claim 2 is characterized in that, Configure the Sink group g1 in Flume-1, which includes Flume-2 with priority 0 and Flume-3 with priority 1; set the processor type to failover and specify the failover timeout parameter. By default, Flume-2 is used as the master node, responsible for writing data to the target Kafka cluster. Flume-3 is used as a standby node, in a standby state, and monitors the running status of Flume-2 in real time.

4. The high-availability data exchange method based on multiple Flume nodes according to claim 3, characterized in that, When Flume-2 fails or becomes unavailable due to downtime, the failover module of Flume-1 detects the abnormality through the heartbeat detection mechanism and automatically switches the data exchange task to Flume-3. After Flume-3 takes over the task, it continues to write data to the target Kafka cluster to ensure that the data exchange process is not interrupted. When Flume-2 resumes operation, the failover module of Flume-1 detects that the master node is available and automatically switches the task back to Flume-2. After Flume-2 takes over the task again, it updates the task status to "success" to ensure the real-time and accuracy of the task status.

5. The highly available data exchange method based on multiple Flume nodes according to claim 4 is characterized in that, For the Flume-1 configuration, define the Sink group g1, which includes Flume-2 and Flume-3, set the processor type to failover, and specify the priority parameters. The priority of Flume-2 is 0, and the priority of Flume-3 is 1. Configure the failover timeout parameter to ensure that it can switch to the standby node when the master node fails. For the Flume-2 configuration, configure the Avro Sink, point to the target Kafka cluster, set a high priority, and execute the data exchange task as the master node. For the Flume-3 configuration, configure the Avro Sink, point to the target Kafka cluster, set a low priority, and be in a standby state as a standby node.

6. The high-availability data exchange method based on multiple Flume nodes according to claim 5, characterized in that, The Sink group mechanism of Flume synchronizes the task status of the master and standby nodes in real time. When the master node recovers, it automatically synchronizes the task status to "success". Set up an integrated monitoring tool to monitor the running status of Flume nodes in real time. When a node failure is detected, trigger an alarm and notify the operation and maintenance personnel to handle it in a timely manner.

7. The high-availability data exchange method based on multiple Flume nodes according to claim 6, characterized in that, Set the global memory parameter in the flume-env.sh configuration file, or dynamically adjust it in the startup command for memory optimization; When performing Flume batch processing, configure the Flume batch processing parameters and write to the target Kafka cluster in batches.

8. High-availability data exchange device based on multiple Flume nodes, characterized in that Including: At least one memory and at least one processor; The at least one memory is used to store machine-readable programs; The at least one processor is used to call the machine-readable program and execute the method described in any one of claims 1 to 7.

Images