A method and apparatus for distributed ETL node fault detection

Abstract

This invention relates to the field of databases, and in particular to a method and apparatus for distributed ETL node fault detection. It mainly includes: when a node acting as the master scheduler starts up, it inserts or updates its own information in the service table of a metadata database; when a node not acting as the master scheduler starts up, it registers itself with the master scheduler, and the master scheduler inserts or updates the corresponding node's information in the service table based on the information registered by the node; the node starts a deadlock fault detection thread, which receives heartbeat anomaly notifications from each node based on the node information recorded in the service table, and restores the normal heartbeat of nodes experiencing network failures. This invention allows each node to use an independent node deadlock fault detection thread to handle whether a node is experiencing a deadlock due to network failure or system overload. Simultaneously, the service table in the metadata database records the node's status, enabling other nodes to obtain fault information.

Description

Technical Field

[0001] This invention relates to the field of databases, and in particular to a method and apparatus for detecting faults in distributed ETL nodes. Background Technology

[0002] Extract-Transform-Load (ETL) is a crucial software tool for data processing and data warehouse construction. It handles the extraction of data from heterogeneous data sources from source databases, cleansing and transforming the data, and loading it into the destination database. Traditional ETL typically deploys the process to a centralized ETL server node. All processes or components within a process utilize multi-threading, meaning even with numerous processes, they can only run on a single node. Furthermore, this approach fails to improve performance for large datasets. Addressing the shortcomings of traditional centralized ETL, distributed ETL has emerged. Distributed ETL generally consists of one or more nodes acting as managers to govern the system's nodes (schedulers), and one or more nodes executing the ETL processes (executors). The scheduler distributes ETL processes to the executors, which then run the ETL processes.

[0003] The biggest difference between distributed ETL and traditional single-machine ETL is that distributed ETL must consider fault tolerance for each node. For example, the scheduler, as the task allocation scheduler of the entire system, must have another node take over the scheduler role if it fails; if an executor fails, the scheduler can no longer schedule processes to run on that executor. A stable node management scheme is necessary to ensure that in a distributed ETL environment, the failure of any node does not affect the operation and monitoring of the entire system. Although the scheduler does not run the specific ETL process, it plays a crucial role in the scheduling, monitoring, and resource management of the ETL process in the entire distributed system. A stable and reliable distributed ETL system must enable a multi-schedule mechanism, that is, one master scheduler and one or more backup schedulers. Since only one master scheduler is allowed in the system, fault detection of the master and backup schedulers becomes a key point and key technology for system stability. Generally, distributed environments use heartbeat detection, such as sending heartbeat data packets to determine the status of nodes. If a node does not receive a heartbeat data packet from the other node within the heartbeat interval, it is determined that the other node is faulty. When the network is unstable or the system is busy, there is a phenomenon of apparent death, which may lead to false judgments.

[0004] Therefore, how to overcome the shortcomings of existing technologies and solve the problem of inconvenient fault detection in distributed ETL environments is a problem to be solved in this technical field. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of existing technologies, the present invention solves the problem of inconvenient fault detection in distributed ETL environments.

[0006] The embodiments of the present invention adopt the following technical solutions:

[0007] In a first aspect, the present invention provides a method for distributed ETL node fault detection, specifically: when the master scheduler starts, it inserts or updates its own information in the service table of the metadata database; when a non-master scheduler starts, it registers itself with the master scheduler, and the master scheduler inserts or updates the information of the corresponding node in the service table according to the information when the node is registered; the node starts a dead fault detection thread, and the dead fault detection thread receives the heartbeat abnormal notification of each node according to the master scheduler and non-master scheduler information recorded in the receiving service table, and restores the normal heartbeat of the node with network fault.

[0008] Preferably, receiving heartbeat anomaly notifications from each node and restoring normal heartbeats for nodes experiencing network failures specifically includes: a non-master scheduler node's dead fault detection thread waiting for the master scheduler and heartbeat anomaly notifications; upon receiving the anomaly notification, accessing the metadata database, and restoring normal heartbeats based on the access results.

[0009] Preferably, restoring normal heartbeat based on the access result specifically includes: if the access is unsuccessful, the node's working status is network fault, and a network fault alarm is issued; if the access is successful, the normal heartbeat between the master scheduler and the non-master scheduler is restored according to the node type.

[0010] Preferably, the node types include: master scheduler, standby scheduler, and executor. The step of restoring normal heartbeat between the master scheduler and non-master scheduler according to the node type specifically includes: when the node is a standby scheduler or executor, obtaining the current master scheduler, notifying the master scheduler whose heartbeat thread is normal, and restoring normal heartbeat between the node and the master scheduler; when the node is the master scheduler, determining whether the master scheduler needs to switch to the standby scheduler, and switching according to the determination result, thereby restoring normal heartbeat between the master scheduler and non-master scheduler.

[0011] Preferably, obtaining the current master scheduler specifically includes: the node's deadlock fault detection thread accessing the service table in the metadata to obtain the master scheduler's information, and accessing the master scheduler via remote service call according to the obtained information; if it can access the master scheduler normally and the master scheduler is working normally, the master scheduler is designated as the current scheduler, and the heartbeat thread is notified; if it cannot access the master scheduler normally, the fault detection thread is put into sleep according to a specified sleep cycle, and attempts to access the master scheduler again at the end of each sleep cycle until it can access the master scheduler normally.

[0012] Preferably, when the node is a standby scheduler, obtaining the current master scheduler further includes: when the obtained current master scheduler is the node itself, determining whether the node has been set as the master scheduler; if it has been set as the master scheduler, setting the current master scheduler to the node itself, and ending the deadlock fault detection thread on the node.

[0013] Preferably, the step of determining whether the primary scheduler needs to be switched to the backup scheduler and switching according to the determination result specifically includes: obtaining node information from the service table and determining whether the node type is a backup scheduler; when the node type is a backup scheduler, switching the node to the backup scheduler, stopping the sending of heartbeat data packets, and starting the backup scheduler's dead fault detection thread; when the node type is the primary scheduler, not switching the node type.

[0014] Preferably, when a node's working state is faulty, after the false death fault detection thread restores a normal heartbeat based on the abnormal heartbeat of the main scheduler, it further includes: the main scheduler scheduling the process to the corresponding executor; when the process starts or ends, the executor notifies the main scheduler of the process's running status; the main scheduler caches the corresponding running instances based on the received running status and generates corresponding running logs in the metadata database, using the running logs to cache information for each process, as well as the parent-child relationships between different processes, and obtains monitoring information consistent with the process running status; the main scheduler determines the fault status of the executor based on the fault monitoring information; when a process on any executor fails, the main scheduler restarts the process on that executor based on the running logs; when the main scheduler detects an executor fault, it processes the process in the executor through the running instances cached by the main scheduler, making the running status of the processed process consistent with the monitoring information; when the main scheduler restarts or switches, it revives all running process instances in all executors through the process running logs, making all process running instances managed by the new main scheduler completely consistent with the main scheduler before the fault.

[0015] Before the master scheduler starts, the process further includes: when the current scheduler starts, and / or when the current scheduler attempts to switch to the master scheduler, the current scheduler creates a master scheduler contention table in the metadata database, using the master scheduler contention table as a distributed mutex lock between different schedulers; after the current scheduler successfully creates the master scheduler contention table and acquires the distributed mutex lock, it determines whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and the existing master scheduler information stored in the metadata database.

[0016] On the other hand, the present invention provides an apparatus for distributed ETL node fault detection, specifically comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the method for distributed ETL node fault detection in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: each node uses an independent fault monitoring thread to handle whether the node is experiencing network failure or system overload causing a false dead state. At the same time, the service table in the metadata database records the node's status, enabling other nodes to obtain fault information. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0019] Figure 1 A flowchart of a method for distributed ETL node fault detection provided in an embodiment of the present invention;

[0020] Figure 2 A flowchart of another method for distributed ETL node fault detection provided in an embodiment of the present invention;

[0021] Figure 3 A flowchart of another method for distributed ETL node fault detection provided in an embodiment of the present invention;

[0022] Figure 4 A flowchart of another method for distributed ETL node fault detection provided in an embodiment of the present invention;

[0023] Figure 5 A flowchart of another method for distributed ETL node fault detection provided in an embodiment of the present invention;

[0024] Figure 6 A flowchart of another method for distributed ETL node fault detection provided in an embodiment of the present invention;

[0025] Figure 7 A flowchart of another method for distributed ETL node fault detection provided in an embodiment of the present invention;

[0026] Figure 8 A flowchart of another method for distributed ETL node fault detection provided in an embodiment of the present invention;

[0027] Figure 9 A flowchart of another method for distributed ETL node fault detection provided in an embodiment of the present invention;

[0028] Figure 10 A schematic diagram of a distributed ETL node fault detection device provided in an embodiment of the present invention;

[0029] The reference numerals in the attached figures are as follows:

[0030] 11: Processor; 12: Memory. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0032] This invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly describe the functional logic relationship of each structural module, and do not limit the specific software and hardware implementation methods.

[0033] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] Example 1:

[0035] This embodiment provides a method for distributed ETL node fault detection, eliminating false alarms about node faults. The service table in the metadata database stores all node information. The executor and scheduler obtain the necessary node information by accessing the SERVICE table. The executor and scheduler activate a network fault detection thread to handle whether a node is experiencing network failure or system overload causing a apparent freeze, thus eliminating false alarms about node faults.

[0036] like Figure 1 As shown in the figure, the specific steps of the distributed ETL node fault detection method provided in this embodiment of the invention are as follows.

[0037] Step 101: When the main scheduler starts, it inserts or updates its own information in the service table of the metadata database.

[0038] The method provided in this embodiment fully utilizes the metadata database in the distributed ETL environment. A service table is created in the metadata database to store all node information. The executor and scheduler obtain the necessary node information by accessing the service table. In actual implementation, the node information stored in the service table can be determined according to actual business needs, typically including: node ID, the address the node service listens on (HOST_ADDRESS), the port the node service listens on (PORT), the node type (primary scheduler, standby scheduler, executor), and the node status (running, stopped). The service table is used by the heartbeat datagram sending and receiving thread and the network fault detection thread.

[0039] Step 102: When a non-master scheduler starts up, it registers itself with the master scheduler. The master scheduler inserts or updates the information of the corresponding node in the service table based on the information when the node is registered.

[0040] When each node starts up, it registers itself with the master scheduler, which then inserts or updates the node's information in the SERVICE table. Similarly, when the master scheduler starts up, it inserts or updates its own node information in the SERVICE table. In practice, the information to be inserted or updated can be determined based on actual business needs, and typically includes: node ID, the address the node service listens on (HOST_ADDRESS), the port the node service listens on (PORT), the node type (master scheduler, standby scheduler, executor), and the node status (running, stopped).

[0041] Step 103: The node starts a fake death fault detection thread. The fake death fault detection thread receives the heartbeat abnormal notification of each node according to the master scheduler and non-master scheduler information recorded in the receiving service table, and restores the normal heartbeat of the node with network failure.

[0042] To prevent the fault detection process from affecting the normal business operations of the nodes, each node has a dedicated fault detection thread. This thread independently checks the heartbeat data packets sent by the main scheduler to determine whether a node is experiencing a network failure or system overload causing a "freeze," thereby eliminating false fault diagnoses and restoring normal heartbeat detection between nodes. Simultaneously, each node needs to write the operational status obtained by the fault detection thread into the SERVICE table for use by other nodes.

[0043] After steps 101-103 provided in this embodiment, the fault detection of the node can be completed.

[0044] like Figure 2 As shown, the fault detection thread can obtain the working status of the node based on the heartbeat datagram of the master scheduler in the following way.

[0045] Step 201: The non-master scheduler node's deadlock fault detection thread waits for the master scheduler and heartbeat anomaly notification.

[0046] If the node is an executor, the executor's fault detection thread waits for the executor to send a network fault detection notification to the master scheduler's heartbeat data receiving thread. This notification typically occurs when the executor has not received a heartbeat datagram from the master scheduler within the heartbeat interval.

[0047] If the node is a standby scheduler, the standby scheduler's fault detection thread waits for the standby scheduler to send a network fault detection notification to the primary scheduler's heartbeat data receiving thread. This notification typically occurs when the standby scheduler has not received a heartbeat datagram from the primary scheduler within the heartbeat interval.

[0048] If the node is the master scheduler, the master scheduler's fault detection thread waits for heartbeat data packets sent by the standby scheduler or executor.

[0049] Step 202: After receiving the abnormal notification, access the metadata database and restore the normal heartbeat based on the access results.

[0050] If the node is an executor, the executor needs to check for network problems in itself or the master scheduler. If no heartbeat data packet is received, it indicates that there may be a network problem in itself or the master scheduler; or if a heartbeat data packet is received from a master scheduler other than the one at the time the executor started, it indicates that the master scheduler may have a network problem.

[0051] If a node is a backup scheduler, it needs to check for network problems in itself or the primary scheduler. If no heartbeat data packet is received, it indicates that there may be a network problem in itself or the primary scheduler; or if a heartbeat data packet is received from a primary scheduler other than the one used when the executor started, it indicates that the primary scheduler may have a network problem.

[0052] If the node is the master scheduler and the master scheduler has not received any heartbeat data packets, it could be that the backup scheduler or executor has not yet been registered and started, it could be a network problem, or it could be that the network has recovered after a failure.

[0053] After steps 201-202 provided in this embodiment, the fault detection of the node can be completed.

[0054] like Figure 3 As shown, in step 202, the working status of the node can be obtained based on the access results in the following way.

[0055] Step 301: If the access fails, the node's working status is network failure, and a network failure alarm is issued.

[0056] After receiving the notification, the executor network fault detection thread accesses the metadata database. If the access fails, it indicates that there is a network fault in the node, and an alarm is issued.

[0057] After receiving the notification, the backup scheduler network fault detection thread accesses the metadata database. If the access fails, it indicates that there is a network fault on the node, and an alarm is issued.

[0058] Step 302: If the access is successful, restore the normal heartbeat between the master scheduler and the non-master scheduler according to the type of node.

[0059] If there is no network failure, the current master scheduler is obtained, and the heartbeat datagram sending and receiving threads of the executor are notified. Thus, the heartbeat between the executor and the master scheduler returns to normal.

[0060] If there is no network failure, the current primary scheduler is obtained, and the heartbeat datagram sending and receiving threads of the backup scheduler are notified. Thus, the heartbeats of both the backup and primary schedulers return to normal.

[0061] After steps 301-302 provided in this embodiment, the determination of the node's working status can be completed.

[0062] like Figure 4 As shown, the fault status of a node can be obtained based on its type in the following ways.

[0063] Step 401: When the node is a standby scheduler or executor, obtain the current master scheduler, notify the master scheduler whose heartbeat thread is normal, and restore normal heartbeat between the node and the master scheduler.

[0064] When the primary scheduler fails, the standby scheduler or executor may receive heartbeat datagrams from a non-primary scheduler. In order for the standby scheduler or executor to distinguish whether the received heartbeat datagram was sent by the primary scheduler, the fault detection thread needs to identify the correct primary scheduler, and the heartbeat datagram is recovered from the obtained primary scheduler.

[0065] Step 402: When the node is the primary scheduler, determine whether the primary scheduler needs to switch to the backup scheduler, and switch according to the determination result, thereby restoring the normal heartbeat between the primary scheduler and the non-primary scheduler.

[0066] If the primary scheduler does not receive any heartbeat data packets, it needs to notify its fault detection thread to perform network fault detection. Upon receiving the notification, the primary scheduler's fault detection thread first accesses the metadata database. If the access fails, a network fault exists, and an alarm is issued. If no network fault exists, it determines whether the primary scheduler needs to be switched to the backup scheduler.

[0067] After steps 401-402 provided in this embodiment, the fault status can be obtained, and corresponding fault handling can be performed according to the fault status.

[0068] like Figure 5 As shown, the acquisition of the master scheduler in step 401 can be completed in the following way.

[0069] Step 501: The node's apparent dead fault detection thread accesses the service table in the metadata to obtain the main scheduler's information, and then accesses the main scheduler through a remote service call based on the obtained information.

[0070] The executor or standby scheduler accesses the SERVICE table in the metadata to obtain the main scheduler information and accesses the main scheduler via remote service calls.

[0071] Step 502: If access is successful and the master scheduler is functioning correctly, set the master scheduler as the current scheduler and notify the heartbeat thread.

[0072] If the access is successful and it is indeed the primary scheduler, return the primary scheduler.

[0073] Step 503: If normal access is not possible, put the fault detection thread into sleep according to the specified sleep cycle, and try to access the main scheduler again at the end of each sleep cycle until normal access is possible.

[0074] If a normal master scheduler cannot be obtained, the system will sleep for a period of time, such as 1 second, and continue to try until a normal master scheduler is obtained.

[0075] After steps 501-503 provided in this embodiment, the correct master scheduler can be obtained.

[0076] Furthermore, when a node is a standby scheduler, that standby scheduler may have already been switched to the primary scheduler. Therefore, as... Figure 6 As shown, obtaining the current master scheduler requires the following steps.

[0077] Step 601: When the current master scheduler is the node itself, determine whether the node has been set as the master scheduler.

[0078] Unlike the executor network failure detection thread which obtains the current primary scheduler, this backup scheduler may have already switched to the primary scheduler.

[0079] Step 602: If it has already been set as the master scheduler, set the current master scheduler to the node itself and terminate the deadlock fault detection thread on the node.

[0080] The master scheduler obtained is itself, at which point the backup scheduler network fault detection thread needs to exit and terminate.

[0081] After steps 601-602 provided in this embodiment, the standby scheduler can obtain the correct primary scheduler.

[0082] In actual implementation, such as Figure 7 As shown, when the primary scheduler fails, the following steps can be used to determine whether the primary scheduler needs to be switched to the backup scheduler, and the switch can be performed based on the determination result.

[0083] Step 701: Obtain node information from the service table and determine whether the node type is a standby scheduler.

[0084] Retrieve your own scheduler information from the SERVICE table in the metadata database.

[0085] Step 702: When the node type is a standby scheduler, switch the node to the standby scheduler, stop sending heartbeat data packets, and start the standby scheduler's dead fault detection thread.

[0086] If the scheduler type in the message has been changed to standby scheduler, it means that a primary scheduler already exists in the system, and the old primary scheduler needs to be switched to the standby scheduler. Stop the datagram sending and receiving threads and the network fault detection thread, stop the flow scheduling function, start the datagram sending and receiving threads of other schedulers and the network fault detection thread of the standby scheduler, and enable the standby scheduler function.

[0087] Step 703: When the node type is the primary scheduler, do not switch the node type.

[0088] Otherwise, the current master scheduler will remain the master scheduler in the system without switching.

[0089] After steps 701-703 provided in this embodiment, the switchover of the faulty master scheduler can be completed.

[0090] The distributed ETL node fault detection method provided in this embodiment has the following beneficial effects:

[0091] 1. Each node has a dedicated network fault detection thread to handle whether the node is experiencing network failure or system overload causing a apparent freeze, eliminating false judgments of node failure and restoring normal heartbeat detection between nodes.

[0092] 2. The SERVICE table in the metadata database stores all node information. The executor and scheduler will obtain the necessary node information by accessing the SERVICE table. The node information includes fields such as node ID, HOST_ADDRESS (the address the node service listens on), PORT (the port the node service listens on), node type (master scheduler, standby scheduler, executor), and node status (running, stopped). The SERVICE table is used by the heartbeat datagram sending and receiving thread and the network fault detection thread.

[0093] Example 2:

[0094] Furthermore, to prevent node failures from affecting the overall system operation, such as... Figure 8 As shown, when a node is in a faulty state, the following fault handling procedure can also be used.

[0095] Step 2101: The master scheduler schedules the process to the corresponding executor. When the process starts or ends, the executor notifies the master scheduler of the process's running status.

[0096] In order for the master scheduler to obtain the running status of the process on each executor node, when the executor process starts or ends, including when the components in the process start or end, it needs to send a running status change event to notify the master scheduler.

[0097] If the executor stops sending events when the master scheduler fails, the executor must continue sending events to the normal master scheduler until the event notification is successful.

[0098] Step 2102: The main scheduler caches the corresponding running instances based on the received running status, generates corresponding running logs in the metadata database, uses the running logs to cache information for each process and the parent-child relationships of different processes, and obtains monitoring information consistent with the running status of the processes.

[0099] In the metadata database, the running status of the process on each node is recorded through runtime logs. The runtime logs include process running logs, specific monitoring information for the process and components, etc. The executor itself does not maintain runtime logs; instead, they are sent to the scheduler for processing. When the main scheduler receives an event, it saves the runtime logs and maintains the specific monitoring information for the process and components to complete the process running monitoring and the revival of the process on the main scheduler.

[0100] Step 2103: When the main scheduler detects an executor failure, it processes the process in the executor through the running instance cached by the main scheduler, so that the running status of the processed process is consistent with the monitoring information. The main scheduler judges the failure status of the executor based on the failure monitoring information. When a process on any executor fails, the main scheduler restarts the process on that executor according to the running log.

[0101] Because the master scheduler and executors have different functions in the ETL process, different fault handling mechanisms are required. When resolving executor failures, the process status stops normally or is rescheduled and restarted. When resolving master scheduler restarts or switches, the new master scheduler revives the running instances of the flow, achieving a completely smooth transition and implementing normal monitoring and management of the process. This approach ensures that the process running status obtained by the master scheduler through monitoring information is completely consistent with the actual running status of the process on the nodes, thereby ensuring that the process is consistent with its state before the failure after restarting or reviving.

[0102] Step 2104: When the master scheduler restarts or switches, revive all running process instances in all executors through the process execution log, so that all process execution instances managed by the new master scheduler are completely consistent with the master scheduler before the failure.

[0103] To address the issue of a smooth transition when the master scheduler restarts or switches over, the new master scheduler revives running instances of the process flow, ensuring normal monitoring and management of the process operation. This approach guarantees that the process running status obtained by the master scheduler through monitoring information is completely consistent with the actual running status of the process on the node, thus ensuring that the process remains consistent with its state before the failure after restarting or reviving.

[0104] After steps 2101-2104 provided in this embodiment, fault tolerance after node failure in the ETL process can be completed.

[0105] The executor can cache information for each process, as well as the parent-child relationships between different processes, based on the received runtime logs in the following ways.

[0106] Step 2201: When the main scheduler schedules a process to the executor for execution, the main scheduler constructs a running instance for each process, as well as the parent-child relationship between each running instance and the running instances of other processes. Each running process corresponds to one running instance.

[0107] The main scheduler caches all running instances (FlowTasks) of processes in memory. One FlowTask corresponds to one ETL process running on the executor.

[0108] Step 2202: When each component in the process starts or ends, the executor sends a corresponding event notification to the main scheduler until the main scheduler successfully receives the event notification.

[0109] Each process may contain multiple components. To ensure the master scheduler can accurately track the execution status of each component, the executor needs to report corresponding event notifications to the master scheduler when each component starts or ends. If the master scheduler fails at this point, the executor must continue sending event notifications to a working master scheduler until a successful notification is received.

[0110] Step 2203: When the main scheduler receives an event notification, it records the received runtime logs to the metadata database through the running instance, and maintains the runtime monitoring information of the process and components in order to monitor the process and revive the running instance.

[0111] When the main scheduler receives a running status event notification from the executor, it promptly writes the running log notification information into the process running log record table of the metadata database through FlowTask, and maintains the specific monitoring information of the process and component running, so as to monitor the running of the process and revive FlowTask.

[0112] After steps 2201-2203 provided in this embodiment, the main scheduler can complete the caching of the running logs.

[0113] The master scheduler can restart the process on the executor based on the runtime log in the following ways.

[0114] Step 2301: The main scheduler obtains the process information on the malfunctioning executor, writes the process information into the metadata database, and adjusts the monitoring information.

[0115] When an executor fails, the master scheduler will no longer receive event notifications for the ETL processes running on that executor. However, the ETL processes running on the failed executor will continue to appear as running in the monitoring information. At this point, the master scheduler needs to proactively handle the processes running on the failed executor, as well as related information about the process component failures, including writing to the metadata database and adjusting the fault monitoring information.

[0116] Step 2302: Based on the process information, obtain the processes currently running on the executor that will fail, schedule the running processes to other executors, and restart the corresponding processes scheduled to other executors.

[0117] If the process restarts due to a fault in the process settings, the scheduler needs to reschedule the process to run on another executor.

[0118] After steps 2301-2302 provided in this embodiment, the process on the faulty actuator can be restarted.

[0119] The following methods can be used to reschedule a process running on a faulty actuator to another actuator and restart the corresponding process.

[0120] Step 2401: Determine whether the process that needs to be restarted has already started running.

[0121] The methods for restarting a process differ depending on whether it has already started or not; therefore, it is first necessary to distinguish whether the process has already started. The process begins execution when the main scheduler receives a start-up event notification from the executor.

[0122] Step 2402: If the process has not started, the main scheduler constructs the start event and fault end event for the process, records the corresponding running log, and adjusts the corresponding parent process.

[0123] Before a process starts running, the main scheduler constructs start and failure end events for the process, writes them to the metadata database log table, adjusts the process running monitoring information, and handles the parent FlowTask events for the process.

[0124] Step 2403: If the process has already started running, obtain all components in the process that started running initially. The main scheduler constructs the failure end events for all components and the failure end events for the process, records the corresponding running logs, and adjusts the corresponding parent processes.

[0125] Once the process has started running, it is necessary to construct component failure termination events and process failure termination events, write them to the metadata database log record table, adjust the process running monitoring information, and handle the parent FlowTask events of the process.

[0126] After steps 2401-2403 provided in this embodiment, it can be ensured that the process operation monitoring information obtained from the main scheduler after the process is restarted is completely consistent with the actual process operation information.

[0127] The following methods can be used to determine whether a process that needs to be restarted has already started running.

[0128] Step 2501: If the main scheduler has received the running status of the process starting, it determines that the process has started running.

[0129] When the main scheduler has received an event notification of the running status, and the status information cached on the main scheduler is consistent with the actual process status, it can be determined that the process has started running.

[0130] Step 2502: When the process on the executor is in the queue and the main scheduler has not received the corresponding running status, it determines that the process has not started running.

[0131] The executor needs to run many processes, some of which are in a queue. The processes in the queue will not send a start event notification to the main scheduler. Before receiving the start event notification, the main scheduler will determine that these processes have not started.

[0132] Step 2503: A fault occurs when the actuator sends the running status to start running, and it is determined that the process has not started running.

[0133] If the executor malfunctions precisely when it is sending the notification event to start the process, the process cannot start normally. In this case, the process should also be considered as not having started.

[0134] After steps 2501-2503 provided in this embodiment, it can be determined whether the process that needs to be restarted has started running.

[0135] In practical implementation, when the distributed ETL master scheduler restarts or switches, if the ETL process is running normally on the executors, the restart or switch of the master scheduler should not stop all processes running on the executors. To handle this situation, the method provided in this embodiment achieves a completely smooth transition by reviving the FlowTask of the master scheduler, and implements normal monitoring and management of process operation.

[0136] The main scheduler can be restarted or switched in the following ways.

[0137] Step 2601: The main scheduler obtains the running logs and filters out the completed running logs.

[0138] After the main scheduler restarts, in order to revive the existing FlowTasks, it is necessary to retrieve the process logs of all running processes from the process execution log record table in the metadata database and filter out the process logs of processes that have already been completed.

[0139] Step 2602: The main scheduler reconstructs the corresponding running instance for each process based on the obtained running logs, configures the parent-child relationship between each running instance, and adjusts the monitoring information.

[0140] To maintain consistency between the FlowTasks on the main scheduler and those before the restart or switchover, for processes that are not running, the FlowTasks for each process need to be reconstructed based on the running process logs, and the parent-child relationships between FlowTasks need to be configured. For processes that are already running, the components of the job process that are currently running and the components that have completed running also need to be reconstructed. The FlowTasks inside the scheduler are completely restored to the state before the main scheduler restarted or switched over.

[0141] Step 2603: The main scheduler constructs monitoring information during the process execution based on the process log and restores the monitoring information to the state before the main scheduler failure.

[0142] To ensure that the monitoring information on the master scheduler is consistent with that before the restart and switchover, it is also necessary to construct process execution monitoring information based on the process logs. The monitoring information is then restored to the state before the master scheduler restarted or switched over.

[0143] After steps 2601-2603 provided in this embodiment, the running instance can be kept consistent with the state before the failure after the main scheduler is restarted or switched.

[0144] Furthermore, during the master scheduler's revival process, it is unable to process the executor's event notifications. Therefore, when the master scheduler reconstructs the running instance, the running status notifications sent by the executor will wait until the master scheduler completes the corresponding process to prevent concurrent conflicts in the process running status settings.

[0145] Example 3:

[0146] Furthermore, in actual implementation, such as Figure 9 As shown, each scheduler node can compete to start the master scheduler before it starts, either when it starts or registers.

[0147] Step 3101: When the current scheduler starts up, and / or when the current scheduler attempts to switch to the primary scheduler, the current scheduler creates a primary scheduler contention table in the metadata database, using the primary scheduler contention table as a distributed mutex lock between different schedulers. When the current scheduler starts up, and / or when the current scheduler attempts to switch to the primary scheduler, the current scheduler creates a scheduling contention table in the metadata database, using the scheduling contention table as a distributed mutex lock between different schedulers.

[0148] In a distributed ETL environment, a metadata database is used to manage the information and storage of the database. In the method provided in this embodiment, the metadata database is at its core, and multiple schedulers compete for the master scheduler through a distributed locking mechanism that uses mutually exclusive table creation within the metadata database.

[0149] The master scheduler contention table (SERVICE_MUTEX) is a distributed mutex table created when the scheduler starts up or when a standby scheduler attempts to switch to the master scheduler, serving as a mutex lock between different schedulers.

[0150] Step 3102: After the current scheduler successfully creates the master scheduler contention table and acquires the distributed mutex lock, it determines whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and the existing master scheduler information stored in the metadata database. If the current scheduler can successfully create the scheduling contention table, it determines whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and its running status.

[0151] Based on the characteristics of mutex locks, each metadata database can only have one scheduling contention table. Therefore, if the current scheduler can successfully create a scheduling contention table, other schedulers cannot create competing scheduling tables, which is equivalent to the current scheduler successfully acquiring a distributed lock. After successful locking, the current scheduler can be considered to have won the competition. In actual implementation, the scheduler that wins the competition cannot be directly set as the master scheduler. It is also necessary to determine whether it is the same scheduler as the currently used master scheduler based on its ID, host address, and port number, and to determine whether a master scheduler switch is needed based on the current master scheduler's working status. In the method provided in this embodiment, each scheduler can only have one unique ID to prevent multiple schedulers with the same ID from registering in the system, ensuring that only one master scheduler is allowed in a system, and avoiding system process scheduling chaos caused by ID conflicts.

[0152] If the current scheduler fails to create the scheduling contention table, it's generally because another scheduler has already created the table, meaning another scheduler has successfully acquired the distributed mutex lock. In this case, the current scheduler goes into sleep mode according to a specified sleep cycle, and at the end of each sleep cycle, it re-attempts to create the primary scheduler contention table until the primary scheduler successfully creates the table and acquires the distributed mutex lock. For example, it might go into sleep mode with a 1-second cycle and continue creating the SERVICE_MUTEX table to acquire the distributed lock until the SERVICE_MUTEX table is successfully created.

[0153] After the scheduler contention is complete, the scheduling contention table created by the current scheduler is deleted to release the distributed mutex lock. Once the current scheduler successfully acquires the lock, other schedulers will wait for the mutex lock to be released, i.e., wait for the SERVICE_MUTEX table to be deleted. After the SERVICE_MUTEX table is deleted, other schedulers can try to create the scheduling contention table again to compete for the next master scheduler position.

[0154] To enable other nodes to access the successfully contested master scheduler node via remote calls, after the current scheduler starts and successfully acquires a distributed lock, the current scheduler will listen for services at the specified host address (HOST_ADDRESS) and port number (PORT) so that other nodes can access the current scheduler.

[0155] After steps 3101-3102 provided in this embodiment, the contention for the master scheduler can be completed.

[0156] After the current scheduler successfully competes for the primary scheduler position, newly registered schedulers can queue up to attempt to create the SERVICE_MUTEX table in order to compete for the primary scheduler position. Existing standby schedulers can also queue up to attempt to create the SERVICE_MUTEX table in order to switch over to the primary scheduler position. The process of attempting to compete or register is the same as steps 101-102, which involves distributed locking through the SERVICE_MUTEX table to determine whether it can become the primary scheduler position, and releasing the distributed lock after a successful attempt.

[0157] In practice, to facilitate unified management of data across distributed nodes, a SERVICE table can be created in the metadata database to manage all system information, including information about each scheduler node.

[0158] The SERVICE table mainly consists of fields such as node ID, the address the node service listens on (HOST_ADDRESS), the port the node service listens on (PORT), the node type (master scheduler, standby scheduler, executor), and the node status (running, stopped). The SERVICE table may contain one master scheduler or multiple standby schedulers, whose status may be stopped or running.

[0159] Multiple schedulers deployed in the system compete for the primary scheduler through a distributed locking mechanism using mutually exclusive table creation in the metadata database. When multiple schedulers are deployed in a distributed ETL system, only one primary scheduler can exist, with the others serving as backup schedulers. Backup schedulers remain idle while the primary scheduler is functioning normally, except when the fault detection thread is working. Only one primary scheduler record is allowed in the SERVICE table of the metadata database; the remaining schedulers must be backup schedulers. The startup order of nodes in a distributed ETL system is uncertain, and it cannot be forced that the backup schedulers start only after the primary scheduler has successfully started. When the primary scheduler fails, backup schedulers attempting to compete for the primary scheduler cannot be required to compete in an orderly manner. Each backup scheduler has an equal opportunity to compete for the primary scheduler position; therefore, a distributed lock is necessary to mutually exclude the initial registration of multiple schedulers or backup schedulers attempting to switch over to the primary scheduler.

[0160] After successful distributed locking, the SERVICE table in the metadata database is accessed to retrieve the primary scheduler for storage management. If no primary scheduler exists, the current scheduler becomes the primary scheduler, and the SERVICE information for this node is inserted into the metadata database. If a primary scheduler already exists, it is necessary to determine whether the primary scheduler in the SERVICE table is a valid primary scheduler.

[0161] To determine whether the master scheduler stored in the SERVICE table is functioning correctly, the key characteristic of the master scheduler is to distinguish whether the master scheduler ID stored in the SERVICE table is the same as the current scheduler ID.

[0162] After the node database is recorded in the SERVICE table, the current scheduler can be preliminarily determined as the master scheduler based on the data in the SERVICE table. If the current scheduler is not the master scheduler, it then creates a scheduling contention table in the metadata database. Specifically, when the current scheduler successfully requests to create the master scheduler contention table and acquires the distributed mutex lock, if there is no master scheduler in the scheduler information stored in the metadata database, the current scheduler becomes the master scheduler, and the scheduler information stored in the metadata database is updated accordingly.

[0163] In practice, the current scheduler's ID, host address, and port number can be used to determine whether the current scheduler is an existing master scheduler. If the current scheduler is not an existing master scheduler, the current scheduler uses a mutual exclusion method to request the creation of a scheduling contention table in the metadata database.

[0164] In specific scenarios, the competition and setup of the master scheduler can be accomplished in the following ways.

[0165] Step 3201: If the ID of the current scheduler is the same as the ID of the existing master scheduler, the host address of the current scheduler is the same as the host address of the existing master scheduler, and the port number of the current scheduler is the same as the port number of the existing master scheduler, set the current scheduler as the master scheduler and update the information of the master scheduler to the information of the current scheduler.

[0166] If the master scheduler ID stored in the SERVICE table is the same as the current scheduler ID that is competing for the same position, and the fields HOST_ADDRESS and PORT in the record are the same as the current scheduler's HOST_ADDRESS and PORT, it means that the master scheduler in the SERVICE table is definitely in a stopped state. This is because it is impossible for the same HOST_ADDRESS and PORT to be listening for services on two nodes. At this time, the current scheduler becomes the master scheduler, and the current scheduler information in the SERVICE table is updated.

[0167] When the master scheduler ID stored in the SERVICE table is different from the current scheduler ID that is competing for the same position, if the fields HOST_ADDRESS and PORT in the record are the same as the current scheduler's HOST_ADDRESS and PORT, it means that the master scheduler in the SERVICE table is definitely in a stopped state, because it is impossible for the same HOST_ADDRESS and PORT to be listening for services on two nodes. At this time, the current scheduler becomes the master scheduler, the existing master scheduler in the SERVICE table is updated to the standby scheduler, and the current scheduler's SERVICE table field record is inserted.

[0168] Step 3202: If the host address of the current scheduler is different from the host address of the existing master scheduler, and / or the port number of the current scheduler is different from the port number of the existing master scheduler, determine the running status of the existing master scheduler and configure the master scheduler according to the running status.

[0169] If HOST_ADDRESS and PORT are not the same, it means that the current scheduler and the master scheduler are different. Attempt to connect to the master scheduler in the SERVICE table through remote service access. Determine the running status of the master scheduler based on the connection result, and then determine whether it is necessary to update the master scheduler stored in the SERVICE table to the current scheduler based on the running status.

[0170] After steps 3201-3202 provided in this embodiment, the contention for the master scheduler can be completed.

[0171] In practice, the main scheduler can be configured according to its operating status in the following ways.

[0172] Step 3301: Send at least one network response request to the existing master scheduler to determine whether the existing master scheduler can respond normally.

[0173] The running status of the existing master scheduler is determined based on the response results, and settings are configured accordingly. In practice, the PING command can be used to test whether the master scheduler can respond normally. The current scheduler sends a PING command to the master scheduler according to the HOST_ADDRESS and PORT in the SERVICE table. If the PING command receives a normal response, it indicates that the master scheduler can respond normally; otherwise, it indicates that the master scheduler cannot respond normally. Furthermore, to ensure that the node's status is obtained normally, the number of PING attempts can be increased. If multiple PING attempts fail, a fault is determined.

[0174] Step 3302: If the system cannot respond normally, set the current scheduler as the master scheduler and update the master scheduler information to the information of the current scheduler.

[0175] If the master scheduler ID stored in the SERVICE table is the same as the current scheduler ID, but the HOST_ADDRESS and PORT are not the same, the master scheduler in the SERVICE table is accessed via remote service and PING is performed. If the PING fails, it means that the master scheduler in the SERVICE table is broken, the current scheduler becomes the master scheduler, and the current scheduler information in the SERVICE table is updated.

[0176] If the master scheduler ID stored in the SERVICE table is different from the current scheduler ID, and the HOST_ADDRESS and PORT are not the same, the master scheduler in the SERVICE table is accessed via remote service and PING is performed. If the PING fails, it means that the master scheduler in the SERVICE table has failed, and the current scheduler becomes the master scheduler. Similarly, the old scheduler that already exists in the SERVICE table is updated to the standby scheduler, and the current scheduler's SERVICE table field record is inserted.

[0177] Step 3303: If a normal response is achieved, and the ID of the current scheduler is the same as the ID of the existing master scheduler, modify the ID of the current scheduler and then re-compete for the master scheduler.

[0178] In other cases, if the master scheduler in the metadata database is normal, but has the same ID as the current scheduler, the current scheduler needs to report an error, exit the program, modify the configuration to adjust the ID, and then register again.

[0179] Step 3304: If a normal response is achieved, and the ID of the current scheduler is different from the ID of the existing primary scheduler, the current scheduler is designated as the backup scheduler, and the information of the existing primary scheduler is not updated.

[0180] In other cases, if the primary scheduler in the metadata database is normal, the current scheduler can only become the standby scheduler. In this case, the standby scheduler registers with the primary scheduler and inserts the standby scheduler's SERVICE table field record information into the metadata database.

[0181] After completing steps 3301-3304 provided in this embodiment, the main scheduler can be set up.

[0182] In practical implementation scenarios, when the scheduler starts up, restarts, or the backup scheduler detects a failure in the primary scheduler via heartbeat, and the primary and backup schedulers compete to become the primary scheduler, the complete process of the primary scheduler competition method provided in this embodiment is as follows.

[0183] Step 3401: Participating nodes create the SERVICE_MUTEX table in the system metadata database. Once the SERVICE_MUTEX table is successfully created and the distributed mutex lock is acquired, the node determines whether it can become the master scheduler through the following steps. If creation fails, the node sleeps for a certain period of time and then tries to create the table again until it is successfully created.

[0184] Step 3402: When the scheduler starts or restarts, it listens for services at a specified address and port so that other nodes can access this node.

[0185] Step 3403: Access the SERVICE table in the system metadata database, which manages all nodes of the system, including all primary and standby schedulers and executors.

[0186] Step 3404: If the master scheduler does not exist in the SERVICE table, then the scheduler whose table was successfully created becomes the master scheduler in the system, and the new master scheduler is written into the SERVICE table.

[0187] Step 3405: ​​If a primary scheduler exists in the SERVICE table, it may be running normally or it may have failed. Therefore, a judgment and PING operation are needed to analyze whether the primary scheduler in the table is a failed primary scheduler. If it is a failed primary scheduler, the node that successfully acquires the lock becomes the primary scheduler, and this is written into the SERVICE table. The failed primary scheduler in the SERVICE table is then changed to a standby scheduler. If the primary scheduler existing in the SERVICE table is a normal primary scheduler, the scheduler that successfully acquires the lock becomes the standby scheduler, and this is also written into the SERVICE table.

[0188] Step 3406: After completing the judgment and settings in step 405, delete the SERVICE_MUTEX table and release the distributed mutex lock.

[0189] Step 3407: Other schedulers queuing for locks execute steps 3401-3406.

[0190] After steps 3401-3407 provided in this embodiment, the competition for the master scheduler among all queuing nodes can be completed.

[0191] Example 4:

[0192] Based on the distributed ETL node fault detection methods provided in Embodiments 1 to 3 above, the present invention also provides an apparatus for implementing the distributed ETL node fault detection methods described above, such as... Figure 10 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The distributed ETL node fault detection device of this embodiment includes one or more processors 11 and a memory 12. Figure 10 Take a processor 11 as an example.

[0193] Processor 11 and memory 12 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0194] The memory 12 serves as a non-volatile computer-readable storage medium for a distributed ETL node fault detection method. It can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the distributed ETL node fault detection methods in Embodiments 1 to 3. The processor 11 executes various functional applications and data processing of the distributed ETL node fault detection apparatus by running the non-volatile software programs, instructions, and modules stored in the memory 12, thereby implementing the distributed ETL node fault detection methods of Embodiments 1 to 3.

[0195] Memory 12 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 12 may optionally include memory remotely located relative to processor 11, which can be connected to processor 11 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0196] The program instructions / modules are stored in memory 12. When executed by one or more processors 11, they perform the distributed ETL node fault detection methods described in Embodiments 1 to 3 above, for example, executing the methods described above. Figures 1 to 9 The steps shown.

[0197] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0198] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for distributed ETL node fault detection, characterized in that, include: When the main scheduler starts, it inserts or updates its own information in the service table of the metadata database; When a non-master scheduler starts up, it registers itself with the master scheduler. The master scheduler then inserts or updates the information of the corresponding node in the service table based on the information registered by the node. A node initiates a deadlock fault detection thread. This thread receives heartbeat anomaly notifications from each node based on the master scheduler and non-master scheduler information recorded in the service table, and restores normal heartbeats for nodes experiencing network failures. This includes: the deadlock fault detection thread of a non-master scheduler node waits for a heartbeat anomaly notification from the master scheduler. Upon receiving the notification, it accesses the metadata database. If the access fails, the node's working status is determined to be network failure, and a network failure alarm is issued. If the access is successful, when the node is a standby scheduler or executor, it obtains the current master scheduler, notifies the master scheduler whose heartbeat thread is functioning normally, and restores normal heartbeats between the node and the master scheduler. When the node is the master scheduler, it determines whether the master scheduler needs to switch to a standby scheduler and switches accordingly, thereby restoring normal heartbeats between the master scheduler and non-master schedulers. Each node uses an independent thread for detecting apparent death.

2. The method for distributed ETL node fault detection according to claim 1, characterized in that, The step of obtaining the current master scheduler specifically includes: The node's apparent death fault detection thread accesses the service table in the metadata to obtain information about the main scheduler, and then accesses the main scheduler through remote service calls based on the obtained information. If access is successful and the master scheduler is functioning correctly, set the master scheduler as the current scheduler and notify the heartbeat thread. If normal access is not possible, the fault detection thread will be put to sleep according to the specified sleep cycle, and will try to access the main scheduler again at the end of each sleep cycle until normal access is possible.

3. The method for distributed ETL node fault detection according to claim 1, characterized in that, When the node is a standby scheduler, obtaining the current primary scheduler further includes: When the current master scheduler is the node itself, determine whether the node has been set as the master scheduler; If it has already been set as the master scheduler, set the current master scheduler to the node itself and terminate the deadlock fault detection thread on the node.

4. The method for distributed ETL node fault detection according to claim 1, characterized in that, The process of determining whether the primary scheduler needs to be switched to the backup scheduler, and switching based on the determination result, specifically includes: Retrieve node information from the service table and determine whether the node type is a standby scheduler; When the node type is standby scheduler, switch the node to standby scheduler, stop sending heartbeat data packets, and start the standby scheduler's dead fault detection thread; When the node type is the primary scheduler, the node type is not switched.

5. The method for distributed ETL node fault detection according to claim 1, characterized in that, When a node's working state is faulty, after restoring the normal heartbeat of the node with network failure, the process further includes: The master scheduler schedules the process to the corresponding executor. When the process starts or ends, the executor notifies the master scheduler of the process's running status. The master scheduler generates corresponding operation logs based on the received operation status, and uses the operation logs to cache information for each process, as well as the parent-child relationships between different processes; When a process on any executor fails, the master scheduler restarts the process on that executor based on the runtime log.

6. The method for distributed ETL node fault detection according to claim 1, characterized in that, Before the main scheduler starts, it also includes: When the current scheduler starts up, and / or when the current scheduler attempts to switch to the primary scheduler, the current scheduler creates a scheduling contention table in the source database, where the scheduling management tables created by each scheduler are mutually exclusive. If the current scheduler can successfully create a scheduling contention table, determine whether the current scheduler can be set as the master scheduler based on the current scheduler's information and running status.

7. A device for distributed ETL node fault detection, characterized in that: The method includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to perform the method for distributed ETL node fault detection as described in any one of claims 1-6.

Images