Method and apparatus for ETL cluster node fault tolerance

Abstract

The present application relates to the field of database, in particular to a kind of ETL cluster node fault tolerance method and device. Including: main dispatcher is scheduled to corresponding executor, and the running state of process is informed to main dispatcher by executor;Main dispatcher caches corresponding running instance according to the running state received, and generates corresponding running log in metadatabase, and the parent-child relationship of different processes, and obtains the monitoring information consistent with process running state;When main dispatcher detects the failure of executor, the process in the executor is handled by the running instance cached by main dispatcher;When main dispatcher restarts or main dispatcher switches, all running process instances in the executor are revived by process running log.The present application: when node fails, the current running state of process on node is recovered by main dispatcher, to realize process restart;After main dispatcher is revived or switches, all node processes managed by it are recovered, to realize the smooth transition of main dispatcher.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of database, in particular to a method and device for fault tolerance of ETL cluster node. BACKGROUND

[0002] Extract, transform, load (ETL) is an important tool software for data processing and building data warehouse, which completes the process of extracting from source database, cleaning and transforming data, and loading to target database of heterogeneous data source. The traditional ETL generally publishes the process to a centralized ETL server node for running, and the running of all processes or components within the process adopts a multi-thread mechanism. More processes can only run on a single node, and a large data processing process cannot improve the performance of data processing. Based on the shortcomings of the traditional centralized ETL, distributed ETL cluster stands out. The distributed ETL cluster generally consists of one or more nodes acting as managers to manage the nodes in the system (schedulers), and one or more nodes executing ETL processes (executors). The scheduler distributes the ETL process to the executor, and the executor runs the ETL process. Unlike the traditional single ETL, the distributed ETL cluster must consider the fault tolerance of each node. For example, once the scheduler, as the task allocation scheduler of the entire system, is down, another node must take over the role of the scheduler; if an executor fails, the scheduler cannot schedule the process to run on the executor. A stable node management solution must be ensured to ensure that the failure of any node in the distributed ETL environment does not affect the operation and monitoring of the entire system. Although the scheduler does not run specific ETL processes, it plays a crucial role in ETL process scheduling, monitoring, and resource management in the entire distributed system. A stable and reliable distributed ETL system must enable multiple scheduler mechanisms, that is, a primary scheduler and one or more standby schedulers. Since only one primary scheduler is allowed in the system, the failure detection of the primary and standby schedulers is a key point and key technology for system stability. Generally, the distributed environment uses heartbeat detection, such as sending heartbeat data packets, to determine the status of the node. If a node does not receive the heartbeat data packet from the other node within the heartbeat interval, it is determined that the other node has failed.

[0003] The fault detection of the scheduler and the executor is ultimately to ensure the stability of the system ETL process. At the same time, there is a problem that the process stops running due to failure, and the monitoring shows a running state that does not match the actual state, or the node fails and the system cannot schedule the process to run, and the entire system is paralyzed. Smooth and stable running of ETL process after failure of distributed ETL cluster node has become a problem that must be solved for distributed ETL cluster.

[0004] In view of this, how to overcome the defects of the prior art, solve the phenomenon of the entire system paralysis caused by a node in the ETL cluster, is a problem to be solved in the technical field. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application solves the problem of the entire system paralysis caused by a node in the ETL cluster.

[0006] The embodiment of the present application adopts the following technical scheme:

[0007] In a first aspect, the present application provides a method for fault tolerance of ETL cluster nodes, specifically: the master scheduler schedules the process to the corresponding executor, and the executor notifies the master scheduler of the running state of the process when the process starts or ends; the master scheduler caches the corresponding running instance according to the received running state, and generates the corresponding running log in the meta database, uses the running log to cache the information of each process, and the parent-child relationship of different processes, and obtains the monitoring information consistent with the process running state; when the master scheduler detects the failure of the executor, the running process in the executor is processed through the running instance cached by the master scheduler, so that the running state of the processed process is consistent with the monitoring information; when the master scheduler restarts or the master scheduler switches, all running process instances in the executor are revived through the process running log, so that all process running instances managed by the new master scheduler are completely consistent with the master scheduler before the failure.

[0008] Preferably, the master scheduler caches the corresponding running instance according to the received running state, and generates the corresponding running log in the meta database, uses the running log to cache the information of each process, and the parent-child relationship of different processes, specifically including: when the master scheduler schedules the process to the executor for execution, the running instance of each process is constructed, and the parent-child relationship of the running instance of each process and the running instance of other processes is constructed, wherein each running process corresponds to a running instance; each component in the process starts or ends, and the executor sends a corresponding event notification to the master scheduler until the master scheduler successfully receives the event notification; when the master scheduler receives the event notification, the received running log is recorded into the meta database through the running instance, and the monitoring information of the runtime of the process and the component is maintained, so as to perform the running monitoring of the process and the revival of the running instance.

[0009] Preferably, the process executed by the running instance cached by the master scheduler in the executor is executed, and specifically includes: the master scheduler acquires process information on the failed executor, writes the process information into a metadata database, and adjusts monitoring information; acquires a running process on the failed executor according to the process information, schedules the running process to other executors, and restarts the process scheduled to the other executors.

[0010] Preferably, the process of scheduling the running process to other executors and restarting the process scheduled to the other executors specifically includes: determining whether the process to be restarted has started running; if the process has not started running, the master scheduler constructs a start event and a failure end event of the process running time, records corresponding running logs, and adjusts a corresponding parent process; if the process has started running, all components in the process that have started running are acquired, the master scheduler constructs failure end events of all the components and a failure end event of the process, records corresponding running logs, and adjusts a corresponding parent process.

[0011] Preferably, the process of determining whether the process to be restarted has started running specifically includes: if the master scheduler has received a running state of the process starting running, it is determined that the process has started running; when the process on the executor is in a queue, the master scheduler does not receive a corresponding running state, and it is determined that the process has not started running; when the executor sends a running state of starting running, a failure occurs, and it is determined that the process has not started running.

[0012] Preferably, when the master scheduler is restarted or the master scheduler is switched, all running process instances in the executors are revived through the process running logs, and specifically includes: the master scheduler acquires running logs that are running, and filters out completed running logs; the master scheduler re-constructs corresponding running instances for each process acquired according to the acquired running logs, configures parent-child relationships between each running instance, and adjusts monitoring information; the master scheduler constructs monitoring information of the process running time according to the running process logs, and restores the monitoring information to a state before the master scheduler fails.

[0013] Preferably, when the master scheduler is restarted or the master scheduler is switched, all running process instances in the executors are revived through the process running logs, and specifically includes: when the master scheduler re-constructs the running instances, the running state notification sent by the executor is always waiting until the master scheduler completes the corresponding process.

[0014] Preferably, before the main scheduler schedules the flow to the corresponding executor, the method further comprises: when the current scheduler starts, and / or when the current scheduler attempts to switch to the main scheduler, the current scheduler creates a main scheduler competition table in the meta database, taking the main scheduler competition table as a distributed mutual lock between different schedulers; after the current scheduler successfully creates the main scheduler competition table and acquires the distributed mutual lock, the current scheduler determines whether the current scheduler can be set as the main scheduler according to the information of the current scheduler and the information of the existing main scheduler stored in the meta database.

[0015] Preferably, the main scheduler determines the fault state of the executor according to the monitoring information of the fault, and specifically comprises: when the main scheduler starts, inserting or updating the information of the main scheduler in the service table of the meta database; when the non-main scheduler starts, registering the non-main scheduler to the main scheduler, and the main scheduler inserts or updates the information of the corresponding node in the service table according to the information of the node when the node registers; the node starts a false death fault detection thread, and the false death fault detection thread receives the heartbeat exception notification of each node according to the main scheduler and the non-main scheduler information recorded in the service table, and restores the normal heartbeat of the node with network fault.

[0016] In another aspect, the application provides an apparatus for ETL cluster node fault tolerance, specifically comprising: at least one processor and a memory, the at least one processor and the memory are connected through a data bus, the memory stores instructions executable by the at least one processor, and the instructions are used to complete the method for ETL cluster node fault tolerance in the first aspect after being executed by the processor.

[0017] Compared with the prior art, the application has the beneficial effects that: the node as the executor notifies the main scheduler of the flow running state, the main scheduler records the running state of the flow on each node through the running log, when the node fails, the main scheduler restores the current running state of the flow on the node according to the running log, so as to realize the restart of the flow on the fault node.

[0018] Further, in the preferred scheme of the application, when the main scheduler fails, the main scheduler is revived or switched, and then the running log is used to restore the flow of all nodes managed by the main scheduler, so that the smooth transition of the main scheduler is realized. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments of the application. Obviously, the drawings described below are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creating any creative labor on the basis of these drawings.

[0020] Figure 1A method flow chart for ETL cluster node fault tolerance provided by an embodiment of the present application is provided.

[0021] Figure 2 A method flow chart for ETL cluster node fault tolerance provided by another embodiment of the present application is provided.

[0022] Figure 3 A method flow chart for ETL cluster node fault tolerance provided by another embodiment of the present application is provided.

[0023] Figure 4 A method flow chart for ETL cluster node fault tolerance provided by another embodiment of the present application is provided.

[0024] Figure 5 A method flow chart for ETL cluster node fault tolerance provided by another embodiment of the present application is provided.

[0025] Figure 6 A method flow chart for ETL cluster node fault tolerance provided by another embodiment of the present application is provided.

[0026] Figure 7 A method flow chart for ETL cluster node fault tolerance provided by another embodiment of the present application is provided.

[0027] Figure 8 A method flow chart for ETL cluster node fault tolerance provided by another embodiment of the present application is provided.

[0028] Figure 9 A device structure schematic diagram for ETL cluster node fault tolerance provided by an embodiment of the present application is provided.

[0029] In the drawings, the reference signs are as follows:

[0030] 11: processor; 12: memory. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0032] The present application is a system architecture of a specific function system, so in the specific embodiments, the functional logical relationship of each structure module is mainly described, and the specific software and hardware implementation is not limited.

[0033] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other. The present application is described in detail below in combination with the drawings and embodiments.

[0034] Example 1:

[0035] This embodiment provides a method for fault tolerance of ETL cluster nodes. It addresses the issue of ensuring the process stops or is rescheduled and restarted normally when an actuator fails; it also ensures normal monitoring and management of process operation when the main scheduler restarts or switches. This ensures that the monitoring information is completely consistent with the actual operation.

[0036] like Figure 1 As shown, the specific steps of the ETL cluster node fault tolerance method provided in this embodiment of the invention are as follows:

[0037] Step 101: The master scheduler schedules the process to the corresponding executor. When the process starts or ends, the executor notifies the master scheduler of the process's running status.

[0038] In order for the master scheduler to obtain the running status of the process on each executor node, when the executor process starts or ends, including when the components in the process start or end, it needs to send a running status change event to notify the master scheduler.

[0039] If the executor stops sending events when the master scheduler fails, the executor must continue sending events to the normal master scheduler until the event notification is successful.

[0040] Step 102: The main scheduler caches the corresponding running instances based on the received running status, generates corresponding running logs in the metadata database, uses the running logs to cache information for each process, as well as the parent-child relationships between different processes, and obtains monitoring information consistent with the running status of the processes.

[0041] In the metadata database, the running status of the process on each node is recorded through runtime logs. The runtime logs include process running logs, specific monitoring information for the process and components, etc. The executor itself does not maintain runtime logs; instead, they are sent to the scheduler for processing. When the main scheduler receives an event, it saves the runtime logs and maintains the specific monitoring information for the process and components to complete the process running monitoring and the revival of the process on the main scheduler.

[0042] Step 103: When the main scheduler detects an executor failure, it processes the process in the executor through the running instance cached by the main scheduler, so that the running status of the processed process is consistent with the monitoring information.

[0043] Because the main scheduler and the executor have different functions in the ETL process, different fault handling mechanisms are required. When resolving an executor failure, the process status stops normally or is rescheduled and restarted.

[0044] Step 104: When the master scheduler restarts or switches, revive all running process instances in all executors through the process execution log, so that all process execution instances managed by the new master scheduler are completely consistent with the master scheduler before the failure.

[0045] To address the issue of a smooth transition when the master scheduler restarts or switches over, the new master scheduler revives running instances of the process flow, ensuring normal monitoring and management of the process operation. This approach guarantees that the process running status obtained by the master scheduler through monitoring information is completely consistent with the actual running status of the process on the node, thus ensuring that the process remains consistent with its state before the failure after restarting or reviving.

[0046] After steps 101-104 provided in this embodiment, fault tolerance after node failure in the ETL process can be completed.

[0047] like Figure 2 As shown, the executor can cache information for each process, as well as the parent-child relationships between different processes, based on the received runtime logs in the following ways.

[0048] Step 201: When the main scheduler schedules a process to an executor for execution, it establishes the parent-child relationship between each process's running instance and the running instances of other processes, where each running process corresponds to one running instance.

[0049] The main scheduler caches all running instances (FlowTasks) of processes in memory. One FlowTask corresponds to one ETL process running on the executor.

[0050] Step 202: When each component in the process starts or ends, the executor sends a corresponding event notification to the main scheduler until the main scheduler successfully receives the event notification.

[0051] Each process may contain multiple components. To ensure the master scheduler can accurately track the execution status of each component, the executor needs to report corresponding event notifications to the master scheduler when each component starts or ends. If the master scheduler fails at this point, the executor must continue sending event notifications to a working master scheduler until a successful notification is received.

[0052] Step 203: When the main scheduler receives an event notification, it records the received runtime logs to the metadata database through the running instance, and maintains the runtime monitoring information of the process and components in order to monitor the process and revive the running instance.

[0053] When the main scheduler receives a running status event notification from the executor, it promptly writes the running log notification information into the process running log record table of the metadata database through FlowTask, and maintains the specific monitoring information of the process and component running, so as to monitor the running of the process and revive FlowTask.

[0054] After steps 201-202 provided in this embodiment, the main scheduler can complete the caching of the running logs.

[0055] like Figure 3 As shown, the master scheduler can restart the process on the executor based on the runtime log in the following ways.

[0056] Step 301: The main scheduler obtains the process information on the malfunctioning executor, writes the process information into the metadata database, and adjusts the monitoring information.

[0057] When an executor fails, the master scheduler will no longer receive event notifications for the ETL processes running on that executor. However, the ETL processes running on the failed executor will continue to appear as running in the monitoring information. At this point, the master scheduler needs to proactively handle the processes running on the failed executor, as well as related information about the process component failures, including writing to the metadata database and adjusting the fault monitoring information.

[0058] Step 302: Based on the process information, obtain the processes currently running on the executor that will fail, schedule the running processes to other executors, and restart the corresponding processes scheduled to other executors.

[0059] If the process restarts due to a configuration failure, the scheduler needs to reschedule the process to run on another executor.

[0060] After steps 301-302 provided in this embodiment, the process on the faulty actuator can be restarted.

[0061] like Figure 4 As shown, the process running on the faulty executor can be scheduled to other executors and the corresponding process can be restarted in the following ways.

[0062] Step 401: Determine if the process that needs to be restarted has already started running.

[0063] The methods for restarting a process differ depending on whether it has already started or not; therefore, it is first necessary to distinguish whether the process has already started. The process begins execution when the main scheduler receives a start-up event notification from the executor.

[0064] Step 402: If the process has not started, the main scheduler constructs the start event and fault end event for the process, records the corresponding running log, and adjusts the corresponding parent process.

[0065] Before a process starts running, the main scheduler constructs start and failure end events for the process, writes them to the metadata database log table, adjusts the process running monitoring information, and handles the parent FlowTask events for the process.

[0066] Step 403: If the process has already started running, obtain all components in the process that started running initially. The main scheduler constructs the failure termination events for all components and the failure termination events for the process, records the corresponding running logs, and adjusts the corresponding parent processes.

[0067] Once the process has started running, it is necessary to construct component failure termination events and process failure termination events, write them to the metadata database log record table, adjust the process running monitoring information, and handle the parent FlowTask events of the process.

[0068] After steps 401-403 provided in this embodiment, it can be ensured that the process operation monitoring information obtained from the main scheduler after the process is restarted is completely consistent with the actual process operation information.

[0069] like Figure 5 As shown, the following methods can be used to determine whether a process that needs to be restarted has already started running.

[0070] Step 501: If the main scheduler has received the running status of the process starting, it determines that the process has started running.

[0071] When the main scheduler has received an event notification of the running status, and the status information cached on the main scheduler is consistent with the actual process status, it can be determined that the process has started running.

[0072] Step 502: When the process on the executor is in the queue and the main scheduler has not received the corresponding running status, it determines that the process has not started running.

[0073] The executor needs to run many processes, some of which are in a queue. The processes in the queue will not send a start event notification to the main scheduler. Before receiving the start event notification, the main scheduler will determine that these processes have not started.

[0074] Step 503: A fault occurs when the actuator sends the running status to start running, and it is determined that the process has not started running.

[0075] If the executor malfunctions precisely when it is sending the notification event to start the process, the process cannot start normally. In this case, the process should also be considered as not having started.

[0076] After steps 501-503 provided in this embodiment, it can be determined whether the process that needs to be restarted has started running.

[0077] In practical implementation, when the distributed ETL master scheduler restarts or switches, if the ETL process is running normally on the executors, the restart or switch of the master scheduler should not stop all processes running on the executors. To handle this situation, the method provided in this embodiment achieves a completely smooth transition by reviving the FlowTask of the master scheduler, and implements normal monitoring and management of process operation.

[0078] like Figure 6 As shown, the main scheduler can be restarted or switched in the following ways.

[0079] Step 601: The main scheduler obtains the running logs and filters out the completed running logs.

[0080] After the main scheduler restarts, in order to revive the existing FlowTasks, it is necessary to retrieve the process logs of all running processes from the process execution log record table in the metadata database and filter out the process logs of processes that have already been completed.

[0081] Step 602: The main scheduler reconstructs the corresponding running instance for each process based on the obtained running logs, configures the parent-child relationship between each running instance, and adjusts the monitoring information.

[0082] To maintain consistency between the FlowTasks on the main scheduler and those before the restart or switchover, for processes that are not running, the FlowTasks for each process need to be reconstructed based on the running process logs, and the parent-child relationships between FlowTasks need to be configured. For processes that are already running, the components of the job process that are currently running and the components that have completed running also need to be reconstructed. The FlowTasks inside the scheduler are completely restored to the state before the main scheduler restarted or switched over.

[0083] Step 603: The main scheduler constructs monitoring information during the process execution based on the process log and restores the monitoring information to the state before the main scheduler failure.

[0084] To ensure that the monitoring information on the master scheduler is consistent with that before the restart and switchover, it is also necessary to construct process execution monitoring information based on the process logs. The monitoring information is then restored to the state before the master scheduler restarted or switched over.

[0085] After steps 601-603 provided in this embodiment, the running instance can be kept consistent with the state before the failure after the main scheduler is restarted or switched.

[0086] Furthermore, during the master scheduler's revival process, it is unable to process the executor's event notifications. Therefore, when the master scheduler reconstructs the running instance, the running status notifications sent by the executor will wait until the master scheduler completes the corresponding process to prevent concurrent conflicts in the process running status settings.

[0087] The fault tolerance method for ETL cluster nodes provided in this embodiment has the following beneficial effects:

[0088] 1. The main scheduler caches all FlowTask instances of running processes in memory. One FlowTask corresponds to one ETL process running on an executor. The FlowTask records the possible parent-child relationships between processes, the executor information where the process is running, and the processing of process running notification events sent by the executor to the main scheduler. This includes timely logging in the metadata database and maintenance of specific monitoring information for process and component operation.

[0089] 2. When the master scheduler restarts or is switched over, the restarted master scheduler or the new master scheduler after the switchover revives the FlowTask of the running process from the process execution log table in the metadata database, thereby taking over the previous master scheduler. All FlowTasks managed by the new master scheduler are completely consistent with those of the master scheduler before the failure, achieving a completely smooth transition.

[0090] 3. When the main scheduler detects an executor failure, it searches for the FlowTask corresponding to the failed executor from all FlowTasks, sets the process execution monitoring information maintained in the FlowTask to "failed," and writes it to the process execution log table in the metadata database, or reschedules the process execution. This ensures that the monitoring information displayed is completely consistent with the actual execution.

[0091] Example 2:

[0092] Furthermore, such as Figure 7 As shown, when the system restarts or a new scheduler registers, the competition for the master scheduler can be won in the following ways.

[0093] Step 2101: When the current scheduler starts up, and / or when the current scheduler attempts to switch to the primary scheduler, the current scheduler creates a primary scheduler contention table in the metadata database, using the primary scheduler contention table as a distributed mutex lock between different schedulers. When the current scheduler starts up, and / or when the current scheduler attempts to switch to the primary scheduler, the current scheduler creates a scheduling contention table in the metadata database, using the scheduling contention table as a distributed mutex lock between different schedulers.

[0094] In a distributed ETL environment, a metadata database is used to manage the information and storage of the database. In the method provided in this embodiment, the metadata database is at its core, and multiple schedulers compete for the master scheduler through a distributed locking mechanism that uses mutually exclusive table creation within the metadata database.

[0095] The master scheduler contention table (SERVICE_MUTEX) is a distributed mutex table created when the scheduler starts up or when a standby scheduler attempts to switch to the master scheduler, serving as a mutex lock between different schedulers.

[0096] Step 2102: After the current scheduler successfully creates the master scheduler contention table and acquires the distributed mutex lock, it determines whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and the existing master scheduler information stored in the metadata database. If the current scheduler can successfully create the scheduling contention table, it determines whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and its running status.

[0097] Based on the characteristics of mutex locks, each metadata database can only have one scheduling contention table. Therefore, if the current scheduler can successfully create a scheduling contention table, other schedulers cannot create competing scheduling tables, which is equivalent to the current scheduler successfully acquiring a distributed lock. After successful locking, the current scheduler can be considered to have won the competition. In actual implementation, the scheduler that wins the competition cannot be directly set as the master scheduler. It is also necessary to determine whether it is the same scheduler as the currently used master scheduler based on its ID, host address, and port number, and to determine whether a master scheduler switch is needed based on the current master scheduler's working status. In the method provided in this embodiment, each scheduler can only have one unique ID to prevent multiple schedulers with the same ID from registering in the system, ensuring that only one master scheduler is allowed in a system, and avoiding system process scheduling chaos caused by ID conflicts.

[0098] If the current scheduler fails to create the scheduling contention table, it's generally because another scheduler has already created the table, meaning another scheduler has successfully acquired the distributed mutex lock. In this case, the current scheduler goes into sleep mode according to a specified sleep cycle, and at the end of each sleep cycle, it re-attempts to create the primary scheduler contention table until the primary scheduler successfully creates the table and acquires the distributed mutex lock. For example, it might go into sleep mode with a 1-second cycle and continue creating the SERVICE_MUTEX table to acquire the distributed lock until the SERVICE_MUTEX table is successfully created.

[0099] After the scheduler contention is complete, the scheduling contention table created by the current scheduler is deleted to release the distributed mutex lock. Once the current scheduler successfully acquires the lock, other schedulers will wait for the mutex lock to be released, i.e., wait for the SERVICE_MUTEX table to be deleted. After the SERVICE_MUTEX table is deleted, other schedulers can try to create the scheduling contention table again to compete for the next master scheduler position.

[0100] To enable other nodes to access the successfully contested master scheduler node via remote calls, after the current scheduler starts and successfully acquires a distributed lock, the current scheduler will listen for services at the specified host address (HOST_ADDRESS) and port number (PORT) so that other nodes can access the current scheduler.

[0101] After steps 2101-2102 provided in this embodiment, the contention for the master scheduler can be completed.

[0102] After the current scheduler successfully competes for the master scheduler position, newly registered schedulers can queue up to attempt to create the SERVICE_MUTEX table in order to compete for the master scheduler position. Existing standby schedulers can also queue up to attempt to create the SERVICE_MUTEX table in order to switch over to the master scheduler position. The process of attempting to compete or register is the same as steps 2101-2102, which involves distributed locking through the SERVICE_MUTEX table to determine whether it can become the master scheduler, and releasing the distributed lock after a successful attempt.

[0103] In practice, to facilitate unified management of data across distributed nodes, a SERVICE table can be created in the metadata database to manage all system information, including information about each scheduler node.

[0104] The SERVICE table mainly consists of fields such as node ID, the address the node service listens on (HOST_ADDRESS), the port the node service listens on (PORT), the node type (master scheduler, standby scheduler, executor), and the node status (running, stopped). The SERVICE table may contain one master scheduler or multiple standby schedulers, whose status may be stopped or running.

[0105] Multiple schedulers deployed in the system compete for the primary scheduler through a distributed locking mechanism using mutually exclusive table creation in the metadata database. When multiple schedulers are deployed in a distributed ETL system, only one primary scheduler can exist, with the others serving as backup schedulers. Backup schedulers remain idle while the primary scheduler is functioning normally, except when the fault detection thread is working. Only one primary scheduler record is allowed in the SERVICE table of the metadata database; the remaining schedulers must be backup schedulers. The startup order of nodes in a distributed ETL system is uncertain, and it cannot be forced that the backup schedulers start only after the primary scheduler has successfully started. When the primary scheduler fails, backup schedulers attempting to compete for the primary scheduler cannot be required to compete in an orderly manner. Each backup scheduler has an equal opportunity to compete for the primary scheduler position; therefore, a distributed lock is necessary to mutually exclude the initial registration of multiple schedulers or backup schedulers attempting to switch over to the primary scheduler.

[0106] After successful distributed locking, the SERVICE table in the metadata database is accessed to retrieve the primary scheduler for storage management. If no primary scheduler exists, the current scheduler becomes the primary scheduler, and the SERVICE information for this node is inserted into the metadata database. If a primary scheduler already exists, it is necessary to determine whether the primary scheduler in the SERVICE table is a valid primary scheduler.

[0107] To determine whether the master scheduler stored in the SERVICE table is functioning correctly, the key characteristic of the master scheduler is to distinguish whether the master scheduler ID stored in the SERVICE table is the same as the current scheduler ID.

[0108] After the node database is recorded in the SERVICE table, the current scheduler can be preliminarily determined as the master scheduler based on the data in the SERVICE table. If the current scheduler is not the master scheduler, it then creates a scheduling contention table in the metadata database. Specifically, when the current scheduler successfully requests to create the master scheduler contention table and acquires the distributed mutex lock, if there is no master scheduler in the scheduler information stored in the metadata database, the current scheduler becomes the master scheduler, and the scheduler information stored in the metadata database is updated accordingly.

[0109] In practice, the current scheduler's ID, host address, and port number can be used to determine whether the current scheduler is an existing master scheduler. If the current scheduler is not an existing master scheduler, the current scheduler uses a mutual exclusion method to request the creation of a scheduling contention table in the metadata database.

[0110] In specific scenarios, the competition and configuration of the master scheduler can be accomplished in the following ways.

[0111] Step 2201: If the ID of the current scheduler is the same as the ID of the existing master scheduler, the host address of the current scheduler is the same as the host address of the existing master scheduler, and the port number of the current scheduler is the same as the port number of the existing master scheduler, set the current scheduler as the master scheduler and update the information of the master scheduler to the information of the current scheduler.

[0112] If the master scheduler ID stored in the SERVICE table is the same as the current scheduler ID that is competing for the same position, and the fields HOST_ADDRESS and PORT in the record are the same as the current scheduler's HOST_ADDRESS and PORT, it means that the master scheduler in the SERVICE table is definitely in a stopped state. This is because it is impossible for the same HOST_ADDRESS and PORT to be listening for services on two nodes. At this time, the current scheduler becomes the master scheduler, and the current scheduler information in the SERVICE table is updated.

[0113] When the master scheduler ID stored in the SERVICE table is different from the current scheduler ID that is competing for the same position, if the fields HOST_ADDRESS and PORT in the record are the same as the current scheduler's HOST_ADDRESS and PORT, it means that the master scheduler in the SERVICE table is definitely in a stopped state, because it is impossible for the same HOST_ADDRESS and PORT to be listening for services on two nodes. At this time, the current scheduler becomes the master scheduler, the existing master scheduler in the SERVICE table is updated to the standby scheduler, and the current scheduler's SERVICE table field record is inserted.

[0114] Step 2202: If the host address of the current scheduler is different from the host address of the existing master scheduler, and / or the port number of the current scheduler is different from the port number of the existing master scheduler, determine the running status of the existing master scheduler and configure the master scheduler according to the running status.

[0115] If HOST_ADDRESS and PORT are not the same, it means that the current scheduler and the master scheduler are different. Attempt to connect to the master scheduler in the SERVICE table through remote service access. Determine the running status of the master scheduler based on the connection result, and then determine whether it is necessary to update the master scheduler stored in the SERVICE table to the current scheduler based on the running status.

[0116] After steps 2201-2202 provided in this embodiment, the contention for the master scheduler can be completed.

[0117] In practice, the main scheduler can be configured according to its operating status in the following ways.

[0118] Step 2301: Send at least one network response request to the existing master scheduler to determine whether the existing master scheduler can respond normally.

[0119] The running status of the existing master scheduler is determined based on the response results, and settings are configured accordingly. In practice, the PING command can be used to test whether the master scheduler can respond normally. The current scheduler sends a PING command to the master scheduler according to the HOST_ADDRESS and PORT in the SERVICE table. If the PING command receives a normal response, it indicates that the master scheduler can respond normally; otherwise, it indicates that the master scheduler cannot respond normally. Furthermore, to ensure that the node's status is obtained normally, the number of PING attempts can be increased. If multiple PING attempts fail, a fault is determined.

[0120] Step 2302: If the system cannot respond normally, set the current scheduler as the master scheduler and update the master scheduler information to the information of the current scheduler.

[0121] If the master scheduler ID stored in the SERVICE table is the same as the current scheduler ID, but the HOST_ADDRESS and PORT are not the same, the master scheduler in the SERVICE table is accessed via remote service and PING is performed. If the PING fails, it means that the master scheduler in the SERVICE table is broken, the current scheduler becomes the master scheduler, and the current scheduler information in the SERVICE table is updated.

[0122] If the master scheduler ID stored in the SERVICE table is different from the current scheduler ID, and the HOST_ADDRESS and PORT are not the same, the master scheduler in the SERVICE table is accessed via remote service and PING is performed. If the PING fails, it means that the master scheduler in the SERVICE table has failed, and the current scheduler becomes the master scheduler. Similarly, the old scheduler that already exists in the SERVICE table is updated to the standby scheduler, and the current scheduler's SERVICE table field record is inserted.

[0123] Step 2303: If a normal response is achieved, and the ID of the current scheduler is the same as the ID of the existing master scheduler, modify the ID of the current scheduler and then re-compete for the master scheduler.

[0124] In other cases, if the master scheduler in the metadata database is normal, but has the same ID as the current scheduler, the current scheduler needs to report an error, exit the program, modify the configuration to adjust the ID, and then register again.

[0125] Step 2304: If a normal response is achieved, and the ID of the current scheduler is different from the ID of the existing primary scheduler, the current scheduler is designated as the backup scheduler, and the information of the existing primary scheduler is not updated.

[0126] In other cases, if the primary scheduler in the metadata database is normal, the current scheduler can only become the standby scheduler. In this case, the standby scheduler registers with the primary scheduler and inserts the standby scheduler's SERVICE table field record information into the metadata database.

[0127] After completing steps 2301-2304 provided in this embodiment, the main scheduler can be set up.

[0128] In practical implementation scenarios, when the scheduler starts up, restarts, or the backup scheduler detects a failure in the primary scheduler via heartbeat, and the primary and backup schedulers compete to become the primary scheduler, the complete process of the primary scheduler competition method provided in this embodiment is as follows.

[0129] Step 2401: Participating nodes create the SERVICE_MUTEX table in the system metadata database. Once the SERVICE_MUTEX table is successfully created and the distributed mutex lock is acquired, the node determines whether it can become the master scheduler through the following steps. If creation fails, the node sleeps for a certain period of time and then tries to create the table again until it is successfully created.

[0130] Step 2402: When the scheduler starts or restarts, it listens for services at a specified address and port so that other nodes can access this node.

[0131] Step 2403: Access the SERVICE table in the system metadata database, which manages all nodes of the system, including all primary and standby schedulers and executors.

[0132] Step 2404: If the master scheduler does not exist in the SERVICE table, then the scheduler whose table was successfully created becomes the master scheduler in the system, and the new master scheduler is written into the SERVICE table.

[0133] Step 2405: If a primary scheduler exists in the SERVICE table, it may be running normally or it may have failed. Therefore, a judgment and PING operation are needed to analyze whether the primary scheduler in the table is a failed primary scheduler. If it is a failed primary scheduler, the node that successfully acquires the lock becomes the primary scheduler, and this is written into the SERVICE table. The failed primary scheduler in the SERVICE table is then changed to a standby scheduler. If the primary scheduler existing in the SERVICE table is a normal primary scheduler, the scheduler that successfully acquires the lock becomes the standby scheduler, and this is written into the SERVICE table.

[0134] Step 2406: After completing the judgment and settings in step 405, delete the SERVICE_MUTEX table and release the distributed mutex lock.

[0135] Step 2407: Other schedulers queuing for locks execute steps 2401-2406.

[0136] After steps 2401-2407 provided in this embodiment, the competition for the master scheduler among all queuing nodes can be completed.

[0137] Example 3:

[0138] Furthermore, such as Figure 8 As shown, in actual implementation, node faults can be detected using the master scheduler in the following ways.

[0139] Step 3101: When the main scheduler starts, it inserts or updates its own information in the service table of the metadata database.

[0140] The method provided in this embodiment fully utilizes the metadata database in the distributed ETL environment. A service table is created in the metadata database to store all node information. The executor and scheduler obtain the necessary node information by accessing the service table. In actual implementation, the node information stored in the service table can be determined according to actual business needs, typically including: node ID, the address the node service listens on (HOST_ADDRESS), the port the node service listens on (PORT), the node type (primary scheduler, standby scheduler, executor), and the node status (running, stopped). The service table is used by the heartbeat datagram sending and receiving thread and the network fault detection thread.

[0141] Step 3102: When a non-master scheduler starts up, it registers itself with the master scheduler. The master scheduler inserts or updates the corresponding node's information in the service table based on the information registered by the node.

[0142] When each node starts up, it registers itself with the master scheduler, which then inserts or updates the node's information in the SERVICE table. Similarly, when the master scheduler starts up, it inserts or updates its own node information in the SERVICE table. In practice, the information to be inserted or updated can be determined based on actual business needs, and typically includes: node ID, the address the node service listens on (HOST_ADDRESS), the port the node service listens on (PORT), the node type (master scheduler, standby scheduler, executor), and the node status (running, stopped).

[0143] Step 3103: The node starts a dead fault detection thread. The dead fault detection thread receives the heartbeat abnormal notification of each node according to the main scheduler and non-main scheduler information recorded in the receiving service table, and restores the normal heartbeat of the node with network failure. The node starts a fault detection thread. The fault detection thread obtains the working status of the node according to the heartbeat data packet of the main scheduler and writes the obtained working status into the service table.

[0144] To prevent the fault detection process from affecting the normal business operations of the nodes, each node has a dedicated fault detection thread. This thread independently checks the heartbeat data packets sent by the main scheduler to determine whether a node is experiencing a network failure or system overload causing a "freeze," thereby eliminating false fault diagnoses and restoring normal heartbeat detection between nodes. Simultaneously, each node needs to write the operational status obtained by the fault detection thread into the SERVICE table for use by other nodes.

[0145] After steps 3101-3103 provided in this embodiment, the fault detection of the node can be completed.

[0146] The fault detection thread can obtain the working status of a node based on the heartbeat datagram from the master scheduler in the following ways.

[0147] Step 3201: The non-master scheduler node's apparent death fault detection thread waits for the master scheduler and the heartbeat anomaly notification. The fault detection thread waits for the heartbeat datagram sent by the master scheduler.

[0148] If the node is an executor, the executor's fault detection thread waits for the executor to send a network fault detection notification to the master scheduler's heartbeat data receiving thread. This notification typically occurs when the executor has not received a heartbeat datagram from the master scheduler within the heartbeat interval.

[0149] If the node is a standby scheduler, the standby scheduler's fault detection thread waits for the standby scheduler to send a network fault detection notification to the primary scheduler's heartbeat data receiving thread. This notification typically occurs when the standby scheduler has not received a heartbeat datagram from the primary scheduler within the heartbeat interval.

[0150] If the node is the master scheduler, the master scheduler's fault detection thread waits for heartbeat data packets sent by the standby scheduler or executor.

[0151] Step 3202: After receiving the exception notification, access the metadata database and restore the normal heartbeat according to the access result. When no heartbeat data packet is received from the master scheduler within the specified interval, and / or when the fault detection thread receives a heartbeat data packet from a non-master scheduler, access the metadata database and obtain the working status of the node according to the access result.

[0152] If the node is an executor, the executor needs to check for network problems in itself or the master scheduler. If no heartbeat data packet is received, it indicates that there may be a network problem in itself or the master scheduler; or if a heartbeat data packet is received from a master scheduler other than the one at the time the executor started, it indicates that the master scheduler may have a network problem.

[0153] If a node is a backup scheduler, it needs to check for network problems in itself or the primary scheduler. If no heartbeat data packet is received, it indicates that there may be a network problem in itself or the primary scheduler; or if a heartbeat data packet is received from a primary scheduler other than the one used when the executor started, it indicates that the primary scheduler may have a network problem.

[0154] If the node is the master scheduler and the master scheduler has not received any heartbeat data packets, it could be that the backup scheduler or executor has not yet been registered and started, it could be a network problem, or it could be that the network has recovered after a failure.

[0155] After steps 3201-3202 provided in this embodiment, the fault detection of the node can be completed.

[0156] In step 3202, the working status of the node can be obtained based on the access results using the following method.

[0157] Step 3301: If the access fails, the node's working status is network failure, and a network failure alarm is issued.

[0158] After receiving the notification, the executor network fault detection thread accesses the metadata database. If the access fails, it indicates that there is a network fault in the node, and an alarm is issued.

[0159] After receiving the notification, the backup scheduler network fault detection thread accesses the metadata database. If the access fails, it indicates that there is a network fault on the node, and an alarm is issued.

[0160] Step 3302: If the access is successful, restore the normal heartbeat between the master scheduler and the non-master scheduler according to the node type. If the access is successful, restore the heartbeat datagrams of the node according to the node type.

[0161] If there is no network failure, the current master scheduler is obtained, and the heartbeat datagram sending and receiving threads of the executor are notified. Thus, the heartbeat between the executor and the master scheduler returns to normal.

[0162] If there is no network failure, the current primary scheduler is obtained, and the heartbeat datagram sending and receiving threads of the backup scheduler are notified. Thus, the heartbeats of both the backup and primary schedulers return to normal.

[0163] After steps 3301-3302 provided in this embodiment, the determination of the node's working status can be completed.

[0164] The fault status of a node can be obtained based on its type in the following ways.

[0165] Step 3401: When the node is a standby scheduler or executor, obtain the current master scheduler, notify the master scheduler whose heartbeat thread is normal, and restore the normal heartbeat between the node and the master scheduler. When the node is a standby scheduler or executor, obtain the current master scheduler and restore the heartbeat datagram between the node and the master scheduler.

[0166] When the primary scheduler fails, the standby scheduler or executor may receive heartbeat datagrams from a non-primary scheduler. In order for the standby scheduler or executor to distinguish whether the received heartbeat datagram was sent by the primary scheduler, the fault detection thread needs to identify the correct primary scheduler, and the heartbeat datagram is recovered from the obtained primary scheduler.

[0167] Step 3402: When the node is the primary scheduler, determine whether the primary scheduler needs to switch to the backup scheduler, and switch according to the determination result, thereby restoring the normal heartbeat between the primary scheduler and the non-primary scheduler.

[0168] If the primary scheduler does not receive any heartbeat data packets, it needs to notify its fault detection thread to perform network fault detection. Upon receiving the notification, the primary scheduler's fault detection thread first accesses the metadata database. If the access fails, a network fault exists, and an alarm is issued. If no network fault exists, it determines whether the primary scheduler needs to be switched to the backup scheduler.

[0169] After steps 3401-3402 provided in this embodiment, the fault status can be obtained, and corresponding fault handling can be performed according to the fault status.

[0170] The master scheduler can be obtained in step 3401 using the following method.

[0171] Step 3501: The node's apparent death fault detection thread accesses the service table in the metadata, obtains the information of the main scheduler, and accesses the main scheduler through remote service call according to the obtained information.

[0172] The executor or standby scheduler accesses the SERVICE table in the metadata to obtain the main scheduler information and accesses the main scheduler via remote service calls.

[0173] Step 3502: If access is successful and the master scheduler is functioning correctly, set the master scheduler as the current scheduler and notify the heartbeat thread.

[0174] If the access is successful and it is indeed the primary scheduler, return this primary scheduler.

[0175] Step 3503: If normal access is not possible, put the fault detection thread into sleep according to the specified sleep cycle, and try to access the main scheduler again at the end of each sleep cycle until normal access is possible.

[0176] If a normal master scheduler cannot be obtained, the system will sleep for a period of time, such as 1 second, and continue to try until a normal master scheduler is obtained.

[0177] After steps 3501-3503 provided in this embodiment, the correct master scheduler can be obtained.

[0178] Furthermore, when a node is a standby scheduler, it may have already been switched to the primary scheduler. Therefore, obtaining the current primary scheduler requires the following steps.

[0179] Step 3601: When the current master scheduler is the node itself, determine whether the node has been set as the master scheduler.

[0180] Unlike the executor network failure detection thread which obtains the current primary scheduler, this backup scheduler may have already switched to the primary scheduler.

[0181] Step 3602: If it has already been set as the master scheduler, set the current master scheduler to the node itself and terminate the deadlock fault detection thread on the node.

[0182] The master scheduler obtained is itself, at which point the backup scheduler network fault detection thread needs to exit and terminate.

[0183] After steps 3601-3602 provided in this embodiment, the standby scheduler can obtain the correct primary scheduler.

[0184] In practice, when the primary scheduler fails, the following steps can be used to determine whether the primary scheduler needs to be switched to the backup scheduler, and the switch can be performed based on the determination result.

[0185] Step 3701: Obtain node information from the service table and determine whether the node type is a standby scheduler.

[0186] Retrieve your own scheduler information from the SERVICE table in the metadata database.

[0187] Step 3702: When the node type is a standby scheduler, switch the node to the standby scheduler, stop sending heartbeat data packets, and start the standby scheduler's dead fault detection thread.

[0188] If the scheduler type in the message has been changed to standby scheduler, it means that a primary scheduler already exists in the system, and the old primary scheduler needs to be switched to the standby scheduler. Stop the datagram sending and receiving threads and the network fault detection thread, stop the flow scheduling function, start the datagram sending and receiving threads of other schedulers and the network fault detection thread of the standby scheduler, and enable the standby scheduler function.

[0189] Step 3703: When the node type is the primary scheduler, do not switch the node type.

[0190] Otherwise, the current master scheduler will remain the master scheduler in the system without switching.

[0191] After steps 3701-3703 provided in this embodiment, the switchover of the faulty master scheduler can be completed.

[0192] Example 4:

[0193] Based on the ETL cluster node fault tolerance methods provided in Embodiments 1 to 3 above, the present invention also provides an apparatus for implementing ETL cluster node fault tolerance methods, such as... Figure 9 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The fault-tolerant ETL cluster node device of this embodiment includes one or more processors 11 and a memory 12. Figure 9 Take a processor 11 as an example.

[0194] Processor 11 and memory 12 can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0195] The memory 12, as a non-volatile computer-readable storage medium for ETL cluster node fault tolerance, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the ETL cluster node fault tolerance methods in Embodiments 1 to 3. The processor 11 executes various functional applications and data processing of the ETL cluster node fault tolerance device by running the non-volatile software programs, instructions, and modules stored in the memory 12, thereby implementing the ETL cluster node fault tolerance methods of Embodiments 1 to 3.

[0196] Memory 12 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 12 may optionally include memory remotely located relative to processor 11, which can be connected to processor 11 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0197] The program instructions / modules are stored in memory 12. When executed by one or more processors 11, they perform the ETL cluster node fault tolerance methods described in Embodiments 1 to 3 above, for example, performing the methods described above. Figures 1 to 8 The steps shown.

[0198] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0199] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for fault tolerance of ETL cluster nodes, characterized in that, include: The master scheduler schedules the process to the corresponding executor. When the process starts or ends, the executor notifies the master scheduler of the process's running status. The main scheduler caches the corresponding running instances based on the received running status and generates corresponding running logs in the metadata database. It uses these running logs to cache information about each process, the parent-child relationships between different processes, and obtains monitoring information consistent with the process running status. This includes: when the main scheduler schedules a process to an executor for execution, it constructs a running instance for each process, and establishes the parent-child relationships between each process's running instance and other process running instances. Each running process corresponds to one running instance. When each component in a process starts or ends, the executor sends a corresponding event notification to the main scheduler until the main scheduler successfully receives the event notification. When the main scheduler receives an event notification, it records the received running logs to the metadata database through the running instances and maintains the runtime monitoring information of processes and components to enable process monitoring and the revival of running instances. When the main scheduler detects an executor failure, it processes the process in the executor through the running instance cached by the main scheduler, so that the running status of the processed process is consistent with the monitoring information. When the master scheduler restarts or switches, all running process instances in all executors are revived through the process execution log, so that all process execution instances managed by the new master scheduler are completely consistent with the master scheduler before the failure.

2. The method for fault tolerance of ETL cluster nodes according to claim 1, characterized in that, The process in the executor that is cached by the main scheduler specifically includes: The main scheduler obtains process information from the malfunctioning executor, writes the process information into the metadata database, and adjusts the monitoring information accordingly. Based on the process information, obtain the processes currently running on the malfunctioning executor, schedule the running processes to other executors, and restart the processes scheduled to other executors.

3. The method for fault tolerance of ETL cluster nodes according to claim 2, characterized in that, The process of rescheduling a running process to another executor and restarting the process rescheduled to another executor specifically includes: Determine if the process that needs to be restarted has already started running; If the process has not started, the main scheduler constructs the start event and fault end event for the process to run, records the corresponding running log, and adjusts the corresponding parent process. If the process has already started running, obtain all components in the initially running process, the main scheduler constructs the failure termination events for all components and the failure termination event for the process, records the corresponding running logs, and adjusts the corresponding parent process.

4. The method for fault tolerance of ETL cluster nodes according to claim 3, characterized in that, The determination of whether the process requiring a restart has already started includes: If the main scheduler has received the running status of the process starting, it determines that the process has started running; When a process on an executor is in the queue and the main scheduler does not receive the corresponding running status, it determines that the process has not started running. A fault occurs when the actuator sends a running status indicating that it has started running, indicating that the process has not started.

5. The method for fault tolerance of ETL cluster nodes according to claim 1, characterized in that, When the master scheduler restarts or switches, the step of reviving all running process instances in all executors through the process execution log specifically includes: The main scheduler retrieves the logs of currently running processes and filters out the logs of completed processes. The main scheduler reconstructs the corresponding running instance for each process based on the obtained running logs, configures the parent-child relationship between each running instance, and adjusts the monitoring information. The master scheduler constructs monitoring information during process execution based on the process logs and restores the monitoring information to the state before the master scheduler failed.

6. The method for fault tolerance of ETL cluster nodes according to claim 1, characterized in that, When the master scheduler restarts or switches, the step of reviving all running process instances in all executors through the process execution log further includes: When the master scheduler reconstructs the running instance, the running status notification sent by the executor waits until the master scheduler completes the corresponding process.

7. The method for fault tolerance of ETL cluster nodes according to claim 1, characterized in that, Before the main scheduler schedules the process to the corresponding executor, it also includes: When the current scheduler starts up, and / or when the current scheduler attempts to switch to the primary scheduler, the current scheduler creates a primary scheduler contention table in the metadata database, using the primary scheduler contention table as a distributed mutex lock between different schedulers. After the current scheduler successfully creates the master scheduler contention table and acquires the distributed mutex lock, it determines whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and the existing master scheduler information stored in the metadata database.

8. The method for fault tolerance of ETL cluster nodes according to claim 1, characterized in that, The method also includes When the main scheduler starts, it inserts or updates its own information in the service table of the metadata database; When a non-master scheduler starts up, it registers itself with the master scheduler. The master scheduler then inserts or updates the corresponding node's information in the service table based on the information registered by the node. The node starts a node dead fault detection thread. Centered on the system metadata database, the dead fault detection thread receives heartbeat abnormal notifications from each node based on the main scheduler and non-main scheduler information recorded in the received service table, and restores the normal heartbeat of nodes with network failures.

9. A fault-tolerant device for ETL cluster nodes, characterized in that: The method includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to perform the fault tolerance method for ETL cluster nodes as described in any one of claims 1-8.

Images