A method and apparatus for distributed ETL master scheduler contention

Abstract

The present application relates to the field of database, in particular to a kind of distributed ETL master scheduler competition method and device.Mainly include: when current scheduler starts, and / or, when current scheduler attempts to switch to master scheduler, current scheduler creates master scheduler competition table in metadata base, to master scheduler competition table as the distributed mutual lock between different schedulers;After current scheduler successfully creates master scheduler competition table and obtains distributed mutual lock, whether current scheduler can be set as master scheduler is determined according to the information of current scheduler and the information of existing master scheduler stored in metadata base.The present application can realize the distributed locking of master scheduler through master scheduler competition table, so as to simply and effectively complete the competition of master scheduler, quickly configure available master scheduler for the ETL process of distributed database, and avoid scheduling conflict in distributed ETL scenario.

Description

Technical Field

[0001] This invention relates to the field of databases, and in particular to a method and apparatus for contention among distributed ETL master schedulers. Background Technology

[0002] Extract-Transform-Load (ETL) is a crucial software tool for data processing and data warehouse construction. It handles the process of extracting data from heterogeneous data sources from a metadata database, cleaning and transforming the data, and loading it into a target database. Traditional ETL typically deploys the process to a centralized ETL server node. All processes or components within a process utilize multi-threading, meaning that even with numerous processes, they can only run on a single node. Furthermore, this approach fails to improve data processing performance for large datasets.

[0003] Addressing the shortcomings of traditional centralized ETL, distributed ETL has emerged as a superior alternative. Distributed ETL typically consists of one or more nodes acting as managers to govern the nodes in the system, and one or more nodes executing the ETL process. The manager node is called the scheduler, and the nodes executing the ETL process are called executors. The scheduler distributes the ETL process to the executors, who then run it. The biggest difference between distributed ETL and traditional single-machine ETL is that distributed ETL must consider fault tolerance for each node. For example, if the scheduler, as the task allocation scheduler for the entire system, fails, another node must take over the scheduler role; if an executor fails, the scheduler can no longer schedule processes to that executor. To achieve fault tolerance, a stable node management scheme is necessary to ensure that the failure of any node in a distributed ETL environment does not affect the operation and monitoring of the entire system. Although the scheduler does not run the specific ETL process, it plays a crucial role in ETL process scheduling, monitoring, and resource management within the entire distributed system. A stable and reliable distributed ETL system must employ a multi-schedule mechanism, namely one primary scheduler and one or more backup schedulers. Since only one master scheduler is allowed in the system, the competition for the master scheduler among multiple schedulers becomes a key point and critical technology for the stability of the system.

[0004] Therefore, how to overcome the shortcomings of existing technologies and solve the problem of the complex contention mode of the existing master scheduler is a problem to be solved in this technical field. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention solves the problem that the existing master scheduler contention method is relatively complex.

[0006] The embodiments of the present invention adopt the following technical solutions:

[0007] In a first aspect, the present invention provides a method for contention among distributed ETL master schedulers, specifically: when the current scheduler starts up, and / or when the current scheduler attempts to switch to the master scheduler, the current scheduler creates a master scheduler contention table in the metadata database, using the master scheduler contention table as a distributed mutex lock between different schedulers; after the current scheduler successfully creates the master scheduler contention table and acquires the distributed mutex lock, it determines whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and the existing master scheduler information stored in the metadata database.

[0008] Preferably, the current scheduler creates a master scheduler contention table in the metadata database, specifically including: when the current scheduler successfully applies to create the master scheduler contention table and acquires the distributed mutex lock, if there is no master scheduler in the scheduler information stored in the metadata database, the current scheduler becomes the master scheduler and updates the scheduler information stored in the metadata database.

[0009] Preferably, the process of the current scheduler creating a master scheduler contention table in the metadata database further includes: if the current scheduler cannot successfully create the master scheduler contention table, the current scheduler is put into hibernation according to a specified hibernation period, and attempts to create the master scheduler contention table again at the end of each hibernation period, until the master scheduler contention table is successfully created and the distributed mutex lock is acquired.

[0010] Preferably, the information of the current scheduler includes an ID, host address, and port number. The step of determining whether the current scheduler can be set as the master scheduler based on the current scheduler's information and the existing master scheduler information stored in the metadata database specifically includes: if the current scheduler's ID matches the ID of an existing master scheduler, the current scheduler's host address matches the host address of an existing master scheduler, and the current scheduler's port number matches the port number of an existing master scheduler, then the current scheduler is set as the master scheduler, and the master scheduler's information is updated to the current scheduler's information; if the current scheduler's host address differs from the host address of an existing master scheduler, and / or the current scheduler's port number differs from the port number of an existing master scheduler, then the running status of the existing master scheduler is determined, and the master scheduler is set according to the running status.

[0011] Preferably, determining the operating status of the existing master scheduler and setting the master scheduler according to the operating status specifically includes: sending at least one network response request to the existing master scheduler to determine whether the existing master scheduler can respond normally; if it cannot respond normally, setting the current scheduler as the master scheduler and updating the master scheduler information to the information of the current scheduler; if it can respond normally, and the ID of the current scheduler is the same as the ID of the existing master scheduler, modifying the ID of the current scheduler and re-competing for the master scheduler; if it can respond normally, and the ID of the current scheduler is different from the ID of the existing master scheduler, setting the current scheduler as the backup scheduler and not updating the information of the existing master scheduler.

[0012] Preferably, the step of starting the current scheduler further includes: after the current scheduler successfully acquires a distributed lock, performing service listening at a specified host address and port number so that other nodes can access the current scheduler.

[0013] Preferably, after determining whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and the existing master scheduler information stored in the metadata database, the method further includes: deleting the scheduling contention table created by the current scheduler to release the distributed mutex lock.

[0014] Preferably, the metadata database also includes a service table. After determining whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and the existing master scheduler information stored in the metadata database, the process further includes: when the master scheduler starts, inserting or updating its own information in the service table of the metadata database; when a non-master scheduler starts, registering itself with the master scheduler, and the master scheduler inserting or updating the corresponding node's information in the service table based on the information when the node registers; and the node starting a node dead fault detection thread, centered on the system metadata database, receiving heartbeat abnormality notifications from each node based on the master scheduler and non-master scheduler information recorded in the received service table, and restoring normal heartbeats for nodes with network faults.

[0015] Preferably, after determining whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and the existing master scheduler information stored in the metadata database, the process further includes: the master scheduler scheduling the process to the corresponding executor; when the process starts or ends, the executor notifies the master scheduler of the process's running status; the master scheduler caches the corresponding running instances based on the received running status and generates corresponding running logs in the metadata database; the running logs are used to cache the information of each process, as well as the parent-child relationships of different processes, and to obtain monitoring information consistent with the process running status; when the master scheduler detects an executor failure, it processes the processes in the executor through the running instances cached by the master scheduler, so that the running status of the processed processes is consistent with the monitoring information; when the master scheduler restarts or switches, it revives all running process instances in all executors through the process running logs, so that all process running instances managed by the new master scheduler are completely consistent with the master scheduler before the failure.

[0016] On the other hand, the present invention provides an apparatus for competition for a distributed ETL master scheduler, specifically comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the method for competition for a distributed ETL master scheduler in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: each participating scheduler creates a distributed mutually exclusive scheduling competition table in the metadata database, and the master scheduler is distributedly locked through the scheduling competition table, thereby completing the competition for the master scheduler simply and effectively, quickly configuring an available master scheduler for the ETL process of the distributed database, and avoiding scheduling conflicts in the distributed ETL scenario. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0019] Figure 1 A flowchart illustrating a method for contention in a distributed ETL master scheduler, as provided in this embodiment of the invention;

[0020] Figure 2 A flowchart illustrating another method for competition among distributed ETL master schedulers provided in this embodiment of the invention;

[0021] Figure 3A flowchart illustrating another method for competition among distributed ETL master schedulers provided in this embodiment of the invention;

[0022] Figure 4 A flowchart illustrating another method for competition among distributed ETL master schedulers provided in this embodiment of the invention;

[0023] Figure 5 A flowchart illustrating another method for competition among distributed ETL master schedulers provided in this embodiment of the invention;

[0024] Figure 6 A schematic diagram of a device structure for competition among distributed ETL master schedulers provided in an embodiment of the present invention;

[0025] The accompanying figure is labeled as follows:

[0026] 11: Processor; 12: Memory. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0028] This invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly describe the functional logic relationship of each structural module, and do not limit the specific software and hardware implementation methods.

[0029] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] Example 1:

[0031] This embodiment provides a method for contention among distributed ETL master schedulers. This method uses a metadata database in a distributed ETL environment as its core, with multiple schedulers competing for the master scheduler through a distributed locking mechanism that uses mutually exclusive table creation within the metadata database.

[0032] like Figure 1 As shown in the figure, the specific steps of the distributed ETL master scheduler contention method provided in this embodiment of the invention are as follows.

[0033] Step 101: When the current scheduler starts up, and / or when the current scheduler attempts to switch to the primary scheduler, the current scheduler creates a primary scheduler contention table in the metadata database, using the primary scheduler contention table as a distributed mutex lock between different schedulers.

[0034] In a distributed ETL environment, a metadata database is used to manage the information and storage of the database. In the method provided in this embodiment, the metadata database is at its core, and multiple schedulers compete for the master scheduler through a distributed locking mechanism that uses mutually exclusive table creation within the metadata database.

[0035] The master scheduler contention table (SERVICE_MUTEX) is a distributed mutex table created when the scheduler starts up or when a standby scheduler attempts to switch to the master scheduler, serving as a mutex lock between different schedulers.

[0036] Step 102: After the current scheduler successfully creates the master scheduler contention table and acquires the distributed mutex lock, it determines whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and the existing master scheduler information stored in the metadata database.

[0037] Based on the characteristics of mutex locks, each metadata database can only have one scheduling contention table. Therefore, if the current scheduler can successfully create a scheduling contention table, other schedulers cannot create competing scheduling tables, which is equivalent to the current scheduler successfully acquiring a distributed lock. After successful locking, the current scheduler can be considered to have won the competition. In actual implementation, the scheduler that wins the competition cannot be directly set as the master scheduler. It is also necessary to determine whether it is the same scheduler as the currently used master scheduler based on its ID, host address, and port number, and to determine whether a master scheduler switch is needed based on the current master scheduler's working status. In the method provided in this embodiment, each scheduler can only have one unique ID to prevent multiple schedulers with the same ID from registering in the system, ensuring that only one master scheduler is allowed in a system, and avoiding system process scheduling chaos caused by ID conflicts.

[0038] If the current scheduler fails to create the scheduling contention table, it's generally because another scheduler has already created the table, meaning another scheduler has successfully acquired the distributed mutex lock. In this case, the current scheduler goes into sleep mode according to a specified sleep cycle, and at the end of each sleep cycle, it attempts to create the primary scheduler's contention table again, until the primary scheduler successfully creates the table and acquires the distributed mutex lock. For example, it might go into sleep mode with a 1-second cycle, and continue creating the SERVICE_MUTEX table and acquiring the distributed lock according to this cycle until the SERVICE_MUTEX table is successfully created.

[0039] After the scheduler contention is complete, the scheduling contention table created by the current scheduler is deleted to release the distributed mutex lock. Once the current scheduler successfully acquires the lock, other schedulers will wait for the mutex lock to be released, i.e., wait for the SERVICE_MUTEX table to be deleted. After the SERVICE_MUTEX table is deleted, other schedulers can try to create the scheduling contention table again to compete for the next master scheduler position.

[0040] To enable other nodes to access the successfully contested master scheduler node via remote calls, after the current scheduler starts and successfully acquires a distributed lock, it begins service listening at a specified host address (HOST_ADDRESS) and port number (PORT) so that other nodes can access the current scheduler.

[0041] After steps 101-102 provided in this embodiment, the contention for the master scheduler can be completed.

[0042] After the current scheduler successfully competes for the primary scheduler position, newly registered schedulers can queue up to attempt to create the SERVICE_MUTEX table in order to compete for the primary scheduler position. Existing standby schedulers can also queue up to attempt to create the SERVICE_MUTEX table in order to switch over to the primary scheduler position. The process of attempting to compete or register is the same as steps 101-102, which involves distributed locking through the SERVICE_MUTEX table to determine whether it can become the primary scheduler position, and releasing the distributed lock after a successful attempt.

[0043] In practice, to facilitate unified management of data across distributed nodes, a SERVICE table can be created in the metadata database to manage all system information, including information about each scheduler node.

[0044] The SERVICE table mainly consists of fields such as node ID, the address the node service listens on (HOST_ADDRESS), the port the node service listens on (PORT), the node type (master scheduler, standby scheduler, executor), and the node status (running, stopped). The SERVICE table may contain one master scheduler or multiple standby schedulers, whose status may be stopped or running.

[0045] Multiple schedulers deployed in the system compete for the primary scheduler through a distributed locking mechanism using mutually exclusive table creation in the metadata database. When multiple schedulers are deployed in a distributed ETL system, only one primary scheduler can exist, with the others serving as backup schedulers. Backup schedulers remain idle while the primary scheduler is functioning normally, except when the fault detection thread is working. Only one primary scheduler record is allowed in the SERVICE table of the metadata database; the remaining schedulers must be backup schedulers. The startup order of nodes in a distributed ETL system is uncertain, and it cannot be forced that the backup schedulers start only after the primary scheduler has successfully started. When the primary scheduler fails, backup schedulers attempting to compete for the primary scheduler cannot be required to compete in an orderly manner. Each backup scheduler has an equal opportunity to compete for the primary scheduler position; therefore, a distributed lock is necessary to mutually exclude the initial registration of multiple schedulers or backup schedulers attempting to switch over to the primary scheduler.

[0046] After successful distributed locking, the SERVICE table in the metadata database is accessed to retrieve the primary scheduler for storage management. If no primary scheduler exists, the current scheduler becomes the primary scheduler, and the SERVICE information for this node is inserted into the metadata database. If a primary scheduler already exists, it is necessary to determine whether the primary scheduler in the SERVICE table is a valid primary scheduler.

[0047] To determine whether the master scheduler stored in the SERVICE table is functioning correctly, the key characteristic of the master scheduler is to distinguish whether the master scheduler ID stored in the SERVICE table is the same as the current scheduler ID.

[0048] After the node database is recorded in the SERVICE table, the current scheduler can be preliminarily determined as the master scheduler based on the data in the SERVICE table. If the current scheduler is not the master scheduler, it then creates a scheduling contention table in the metadata database. Specifically, when the current scheduler successfully requests to create the master scheduler contention table and acquires the distributed mutex lock, if there is no master scheduler in the scheduler information stored in the metadata database, the current scheduler becomes the master scheduler, and the scheduler information stored in the metadata database is updated accordingly.

[0049] In practice, the current scheduler's ID, host address, and port number can be used to determine whether the current scheduler is an existing master scheduler. If the current scheduler is not an existing master scheduler, the current scheduler uses a mutual exclusion method to request the creation of a scheduling contention table in the metadata database.

[0050] In specific scenarios, such as Figure 2 As shown, the competition and setup of the master scheduler can be accomplished in the following ways.

[0051] Step 201: If the ID of the current scheduler is the same as the ID of the existing master scheduler, the host address of the current scheduler is the same as the host address of the existing master scheduler, and the port number of the current scheduler is the same as the port number of the existing master scheduler, set the current scheduler as the master scheduler and update the information of the master scheduler to the information of the current scheduler.

[0052] If the master scheduler ID stored in the SERVICE table is the same as the current scheduler ID that is competing for the same position, and the fields HOST_ADDRESS and PORT in the record are the same as the current scheduler's HOST_ADDRESS and PORT, it means that the master scheduler in the SERVICE table is definitely in a stopped state. This is because it is impossible for the same HOST_ADDRESS and PORT to be listening for services on two nodes. At this time, the current scheduler becomes the master scheduler, and the current scheduler information in the SERVICE table is updated.

[0053] When the master scheduler ID stored in the SERVICE table is different from the current scheduler ID that is competing for the same position, if the fields HOST_ADDRESS and PORT in the record are the same as the current scheduler's HOST_ADDRESS and PORT, it means that the master scheduler in the SERVICE table is definitely in a stopped state, because it is impossible for the same HOST_ADDRESS and PORT to be listening for services on two nodes. At this time, the current scheduler becomes the master scheduler, the existing master scheduler in the SERVICE table is updated to the standby scheduler, and the current scheduler's SERVICE table field record is inserted.

[0054] Step 202: If the host address of the current scheduler is different from the host address of the existing master scheduler, and / or the port number of the current scheduler is different from the port number of the existing master scheduler, determine the running status of the existing master scheduler and configure the master scheduler according to the running status.

[0055] If HOST_ADDRESS and PORT are not the same, it means that the current scheduler and the master scheduler are different. Attempt to connect to the master scheduler in the SERVICE table through remote service access. Determine the running status of the master scheduler based on the connection result, and then determine whether it is necessary to update the master scheduler stored in the SERVICE table to the current scheduler based on the running status.

[0056] After steps 201-202 provided in this embodiment, the contention for the master scheduler can be completed.

[0057] In specific implementation, such as Figure 3 As shown, the main scheduler can be configured according to its running status in the following manner.

[0058] Step 301: Send at least one network response request to the existing master scheduler to determine whether the existing master scheduler can respond normally.

[0059] The running status of the existing master scheduler is determined based on the response results, and settings are configured accordingly. In practice, the PING command can be used to test whether the master scheduler can respond normally. The current scheduler sends a PING command to the master scheduler according to the HOST_ADDRESS and PORT in the SERVICE table. If the PING command receives a normal response, it indicates that the master scheduler can respond normally; otherwise, it indicates that the master scheduler cannot respond normally. Furthermore, to ensure that the node's status is obtained normally, the number of PING attempts can be increased. If multiple PING attempts fail, a fault is determined.

[0060] Step 302: If the system cannot respond normally, set the current scheduler as the master scheduler and update the master scheduler information to the information of the current scheduler.

[0061] If the master scheduler ID stored in the SERVICE table is the same as the current scheduler ID, but the HOST_ADDRESS and PORT are not the same, the master scheduler in the SERVICE table is accessed via remote service and PING is performed. If the PING fails, it means that the master scheduler in the SERVICE table is broken, the current scheduler becomes the master scheduler, and the current scheduler information in the SERVICE table is updated.

[0062] If the master scheduler ID stored in the SERVICE table is different from the current scheduler ID, and the HOST_ADDRESS and PORT are not the same, the master scheduler in the SERVICE table is accessed via remote service and PING is performed. If the PING fails, it means that the master scheduler in the SERVICE table has failed, and the current scheduler becomes the master scheduler. Similarly, the old scheduler that already exists in the SERVICE table is updated to the standby scheduler, and the current scheduler's SERVICE table field record is inserted.

[0063] Step 303: If a normal response is achieved, and the ID of the current scheduler is the same as the ID of the existing master scheduler, modify the ID of the current scheduler and then compete for the master scheduler again.

[0064] In other cases, if the master scheduler in the metadata database is normal, but has the same ID as the current scheduler, the current scheduler needs to report an error, exit the program, modify the configuration to adjust the ID, and then register again.

[0065] Step 304: If a normal response is received and the ID of the current scheduler is different from the ID of the existing primary scheduler, the current scheduler is set as the backup scheduler, and the information of the existing primary scheduler is not updated.

[0066] In other cases, if the primary scheduler in the metadata database is normal, the current scheduler can only become the standby scheduler. In this case, the standby scheduler registers with the primary scheduler and inserts the standby scheduler's SERVICE table field record information into the metadata database.

[0067] After completing steps 301-304 provided in this embodiment, the main scheduler can be set up.

[0068] In practical implementation scenarios, when the scheduler starts up, restarts, or the backup scheduler detects a failure in the primary scheduler via heartbeat, and the primary and backup schedulers compete to become the primary scheduler, the complete process of the primary scheduler competition method provided in this embodiment is as follows.

[0069] Step 401: Participating nodes create the SERVICE_MUTEX table in the system metadata database. Once the SERVICE_MUTEX table is successfully created and the distributed mutex lock is acquired, the node determines whether it can become the master scheduler through the following steps. If creation fails, the node sleeps for a certain period of time and then tries to create the table again until it is successfully created.

[0070] Step 402: When the scheduler starts or restarts, it listens for services at a specified address and port so that other nodes can access this node.

[0071] Step 403: Access the SERVICE table in the system metadata database, which manages all nodes of the system, including all primary and standby schedulers and executors.

[0072] Step 404: If the master scheduler does not exist in the SERVICE table, then the scheduler whose table was successfully created becomes the master scheduler in the system, and the new master scheduler is written into the SERVICE table.

[0073] Step 405: If a primary scheduler exists in the SERVICE table, it may be running normally or it may have failed. Therefore, a judgment and PING operation are needed to analyze whether the primary scheduler in the table is a failed primary scheduler. If it is a failed primary scheduler, the node that successfully acquires the lock becomes the primary scheduler, and this is written into the SERVICE table. The failed primary scheduler in the SERVICE table is then changed to a standby scheduler. If the primary scheduler existing in the SERVICE table is a normal primary scheduler, the scheduler that successfully acquires the lock becomes the standby scheduler, and this is written into the SERVICE table.

[0074] Step 406: After completing the judgment and settings in step 405, delete the SERVICE_MUTEX table and release the distributed mutex lock.

[0075] Step 407: Other schedulers queuing for locks execute steps 401-406.

[0076] After steps 401-407 provided in this embodiment, the competition for the master scheduler among all queuing nodes can be completed.

[0077] The distributed ETL master scheduler contention method provided in this embodiment has the following advantages over existing technologies:

[0078] 1. Centered on the metadata database in the distributed ETL environment, multiple schedulers compete for the master scheduler through a distributed locking mechanism that uses mutually exclusive table creation in the metadata database.

[0079] 2. Create a SERVICE table in the metadata database to manage all nodes of the system, including the scheduler. Node information is uniformly stored in the SERVICE table for management.

[0080] Example 2:

[0081] Furthermore, such as Figure 4 As shown, after completing the master scheduler settings, the service tables in the metadata database can be used to detect node faults using the master scheduler.

[0082] Step 2101: The master scheduler schedules the process to the corresponding executor. When the process starts or ends, the executor notifies the master scheduler of the process's running status.

[0083] In order for the master scheduler to obtain the running status of the process on each executor node, when the executor process starts or ends, including when the components in the process start or end, it needs to send a running status change event to notify the master scheduler.

[0084] If the executor stops sending events when the master scheduler fails, the executor must continue sending events to the normal master scheduler until the event notification is successful.

[0085] Step 2102: The main scheduler caches the corresponding running instances based on the received running status, generates corresponding running logs in the metadata database, uses the running logs to cache information for each process and the parent-child relationships of different processes, and obtains monitoring information consistent with the running status of the processes.

[0086] In the metadata database, the running status of the process on each node is recorded through runtime logs. The runtime logs include process running logs, specific monitoring information for the process and components, etc. The executor itself does not maintain runtime logs; instead, they are sent to the scheduler for processing. When the main scheduler receives an event, it saves the runtime logs and maintains the specific monitoring information for the process and components to complete the process running monitoring and the revival of the process on the main scheduler.

[0087] Step 2103: When the main scheduler detects an executor failure, it processes the process in the executor through the running instance cached by the main scheduler, so that the running status of the processed process is consistent with the monitoring information. The main scheduler judges the failure status of the executor based on the failure monitoring information. When a process on any executor fails, the main scheduler restarts the process on that executor according to the running log.

[0088] Because the master scheduler and executors have different functions in the ETL process, different fault handling mechanisms are required. When an executor fails, the process status stops normally or is rescheduled and restarted. When the master scheduler restarts or switches, the new master scheduler revives the running instances of the flow, achieving a completely smooth transition and implementing normal monitoring and management of the process. This approach ensures that the process running status obtained by the master scheduler through monitoring information is completely consistent with the actual running status of the process on the nodes, thereby ensuring that the process is consistent with its state before the fault after restarting or reviving.

[0089] Step 2104: When the master scheduler restarts or switches, revive all running process instances in all executors through the process execution log, so that all process execution instances managed by the new master scheduler are completely consistent with the master scheduler before the failure.

[0090] To address the issue of a smooth transition when the master scheduler restarts or switches over, the new master scheduler revives running instances of the process flow, ensuring normal monitoring and management of the process operation. This approach guarantees that the process running status obtained by the master scheduler through monitoring information is completely consistent with the actual running status of the process on the node, thus ensuring that the process remains consistent with its state before the failure after restarting or reviving.

[0091] After steps 2101-2104 provided in this embodiment, fault tolerance after node failure in the ETL process can be completed.

[0092] The executor can cache information for each process, as well as the parent-child relationships between different processes, based on the received runtime logs in the following ways.

[0093] Step 2201: When the main scheduler schedules a process to the executor for execution, the main scheduler constructs a running instance for each process, as well as the parent-child relationship between each running instance and the running instances of other processes. Each running process corresponds to one running instance.

[0094] The main scheduler caches all running instances (FlowTasks) of processes in memory. One FlowTask corresponds to one ETL process running on the executor.

[0095] Step 2202: When each component in the process starts or ends, the executor sends a corresponding event notification to the main scheduler until the main scheduler successfully receives the event notification.

[0096] Each process may contain multiple components. To ensure the master scheduler can accurately track the execution status of each component, the executor needs to report corresponding event notifications to the master scheduler when each component starts or ends. If the master scheduler fails at this point, the executor must continue sending event notifications to a working master scheduler until a successful notification is received.

[0097] Step 2203: When the main scheduler receives an event notification, it records the received runtime logs to the metadata database through the running instance, and maintains the runtime monitoring information of the process and components in order to monitor the process and revive the running instance.

[0098] When the main scheduler receives a running status event notification from the executor, it promptly writes the running log notification information into the process running log record table of the metadata database through FlowTask, and maintains the specific monitoring information of the process and component running, so as to monitor the running of the process and revive FlowTask.

[0099] After steps 2201-2202 provided in this embodiment, the main scheduler can complete the caching of the running logs.

[0100] The fault status of a node can be obtained based on its type in the following ways.

[0101] Step 2401: When the node is a standby scheduler or executor, obtain the current master scheduler and restore the heartbeat data packets between the node and the master scheduler.

[0102] When the primary scheduler fails, the standby scheduler or executor may receive heartbeat datagrams from a non-primary scheduler. In order for the standby scheduler or executor to distinguish whether the received heartbeat datagram was sent by the primary scheduler, the fault detection thread needs to identify the correct primary scheduler, and the heartbeat datagram is recovered from the obtained primary scheduler.

[0103] Step 2402: When the node is the primary scheduler, determine whether the primary scheduler needs to be switched to the backup scheduler, and switch according to the determination result, thereby restoring the heartbeat data packets of the primary scheduler.

[0104] If the primary scheduler does not receive any heartbeat data packets, it needs to notify its fault detection thread to perform network fault detection. Upon receiving the notification, the primary scheduler's fault detection thread first accesses the metadata database. If the access fails, a network fault exists, and an alarm is issued. If no network fault exists, it determines whether the primary scheduler needs to be switched to the backup scheduler.

[0105] After steps 2401-2402 provided in this embodiment, the fault status can be obtained, and corresponding fault handling can be performed according to the fault status.

[0106] The master scheduler can be obtained in step 2401 using the following method.

[0107] Step 2501: The fault detection thread accesses the service table in the metadata to obtain the information of the main scheduler, and accesses the main scheduler through remote service calls according to the obtained information.

[0108] The executor or standby scheduler accesses the SERVICE table in the metadata to obtain the main scheduler information and accesses the main scheduler via remote service calls.

[0109] Step 2502: If access is successful and the master scheduler is functioning correctly, set the master scheduler as the current scheduler.

[0110] If the access is successful and it is indeed the primary scheduler, return this primary scheduler.

[0111] Step 2503: If normal access is not possible, put the fault detection thread into sleep according to the specified sleep cycle, and try to access the main scheduler again at the end of each sleep cycle until normal access is possible.

[0112] If a normal master scheduler cannot be obtained, the system will sleep for a period of time, such as 1 second, and continue to try until a normal master scheduler is obtained.

[0113] After steps 2501-2503 provided in this embodiment, the correct master scheduler can be obtained.

[0114] Furthermore, when a node is a standby scheduler, it may have already been switched to the primary scheduler. Therefore, obtaining the current primary scheduler requires the following steps.

[0115] Step 2601: When the current master scheduler is the node itself, determine whether the node has been set as the master scheduler.

[0116] Unlike the executor network failure detection thread which obtains the current primary scheduler, this backup scheduler may have already switched to the primary scheduler.

[0117] Step 2602: If it has already been set as the master scheduler, set the current master scheduler to the node itself and terminate the fault detection thread on the node.

[0118] The master scheduler obtained is itself, at which point the backup scheduler network fault detection thread needs to exit and terminate.

[0119] After steps 2601-2602 provided in this embodiment, the standby scheduler can obtain the correct primary scheduler.

[0120] In practice, when the primary scheduler fails, the following steps can be used to determine whether the primary scheduler needs to be switched to the backup scheduler, and the switch can be performed based on the determination result.

[0121] Step 2701: Obtain node information from the service table and determine whether the node type is a standby scheduler.

[0122] Retrieve your own scheduler information from the SERVICE table in the metadata database.

[0123] Step 2702: When the node type is standby scheduler, switch the node to standby scheduler, stop sending heartbeat data packets, and start the fault detection thread of the standby scheduler.

[0124] If the scheduler type in the message has been changed to standby scheduler, it means that a primary scheduler already exists in the system, and the old primary scheduler needs to be switched to the standby scheduler. Stop the datagram sending and receiving threads and the network fault detection thread, stop the flow scheduling function, start the datagram sending and receiving threads of other schedulers and the network fault detection thread of the standby scheduler, and enable the standby scheduler function.

[0125] Step 2703: When the node type is the primary scheduler, do not switch the node type.

[0126] Otherwise, the current master scheduler will remain the master scheduler in the system without switching.

[0127] After steps 2701-2703 provided in this embodiment, the switching of the faulty master scheduler can be completed.

[0128] Example 3:

[0129] Furthermore, such as Figure 5 As shown, to prevent the system from failing due to node failure, the process can be restarted through the main scheduler after a node failure.

[0130] Step 3101: When the main scheduler starts, it inserts or updates its own information in the service table of the metadata database.

[0131] The method provided in this embodiment fully utilizes the metadata database in the distributed ETL environment. A service table is created in the metadata database to store all node information. The executor and scheduler obtain the necessary node information by accessing the service table. In actual implementation, the node information stored in the service table can be determined according to actual business needs, typically including: node ID, the address the node service listens on (HOST_ADDRESS), the port the node service listens on (PORT), the node type (primary scheduler, standby scheduler, executor), and the node status (running, stopped). The service table is used by the heartbeat datagram sending and receiving thread and the network fault detection thread.

[0132] Step 3102: When a non-master scheduler starts up, it registers itself with the master scheduler. The master scheduler inserts or updates the corresponding node's information in the service table based on the information registered by the node.

[0133] When each node starts up, it registers itself with the master scheduler, which then inserts or updates the node's information in the SERVICE table. Similarly, when the master scheduler starts up, it inserts or updates its own node information in the SERVICE table. In practice, the information to be inserted or updated can be determined based on actual business needs, and typically includes: node ID, the address the node service listens on (HOST_ADDRESS), the port the node service listens on (PORT), the node type (master scheduler, standby scheduler, executor), and the node status (running, stopped).

[0134] Step 3103: The node starts a dead fault detection thread. The dead fault detection thread receives the heartbeat abnormal notification of each node according to the main scheduler and non-main scheduler information recorded in the receiving service table, and restores the normal heartbeat of the node with network failure. The node starts a fault detection thread. The fault detection thread obtains the working status of the node according to the heartbeat data packet of the main scheduler and writes the obtained working status into the service table.

[0135] To prevent the fault detection process from affecting the normal business operations of the nodes, each node has a dedicated fault detection thread. This thread independently checks the heartbeat data packets sent by the main scheduler to determine whether a node is experiencing a network failure or system overload causing a "freeze," thereby eliminating false fault diagnoses and restoring normal heartbeat detection between nodes. Simultaneously, each node needs to write the operational status obtained by the fault detection thread into the SERVICE table for use by other nodes.

[0136] After steps 3101-3103 provided in this embodiment, the fault detection of the node can be completed.

[0137] The fault detection thread can obtain the working status of a node based on the heartbeat datagram from the master scheduler in the following ways.

[0138] Step 3201: The non-master scheduler node's apparent death fault detection thread waits for the master scheduler and the heartbeat anomaly notification. The fault detection thread waits for the heartbeat datagram sent by the master scheduler.

[0139] If the node is an executor, the executor's fault detection thread waits for the executor to send a network fault detection notification to the master scheduler's heartbeat data receiving thread. This notification typically occurs when the executor has not received a heartbeat datagram from the master scheduler within the heartbeat interval.

[0140] If the node is a standby scheduler, the standby scheduler's fault detection thread waits for the standby scheduler to send a network fault detection notification to the primary scheduler's heartbeat data receiving thread. This notification typically occurs when the standby scheduler has not received a heartbeat datagram from the primary scheduler within the heartbeat interval.

[0141] If the node is the master scheduler, the master scheduler's fault detection thread waits for heartbeat data packets sent by the standby scheduler or executor.

[0142] Step 3202: After receiving the exception notification, access the metadata database and restore the normal heartbeat according to the access result. When no heartbeat data packet is received from the master scheduler within the specified interval, and / or when the fault detection thread receives a heartbeat data packet from a non-master scheduler, access the metadata database and obtain the working status of the node according to the access result.

[0143] If the node is an executor, the executor needs to check for network problems in itself or the master scheduler. If no heartbeat data packet is received, it indicates that there may be a network problem in itself or the master scheduler; or if a heartbeat data packet is received from a master scheduler other than the one at the time the executor started, it indicates that the master scheduler may have a network problem.

[0144] If a node is a backup scheduler, it needs to check for network problems in itself or the primary scheduler. If no heartbeat data packet is received, it indicates that there may be a network problem in itself or the primary scheduler; or if a heartbeat data packet is received from a primary scheduler other than the one used when the executor started, it indicates that the primary scheduler may have a network problem.

[0145] If the node is the master scheduler and the master scheduler has not received any heartbeat data packets, it could be that the backup scheduler or executor has not yet been registered and started, it could be a network problem, or it could be that the network has recovered after a failure.

[0146] After steps 3201-3202 provided in this embodiment, the fault detection of the node can be completed.

[0147] In step 3202, the working status of the node can be obtained based on the access results using the following method.

[0148] Step 3301: If the access fails, the node's working status is network failure, and a network failure alarm is issued.

[0149] After receiving the notification, the executor network fault detection thread accesses the metadata database. If the access fails, it indicates that there is a network fault in the node, and an alarm is issued.

[0150] After receiving the notification, the backup scheduler network fault detection thread accesses the metadata database. If the access fails, it indicates that there is a network fault on the node, and an alarm is issued.

[0151] Step 3302: If the access is successful, restore the normal heartbeat between the master scheduler and the non-master scheduler according to the node type. If the access is successful, restore the heartbeat datagrams of the node according to the node type.

[0152] If there is no network failure, the current master scheduler is obtained, and the heartbeat datagram sending and receiving threads of the executor are notified. Thus, the heartbeat between the executor and the master scheduler returns to normal.

[0153] If there is no network failure, the current primary scheduler is obtained, and the heartbeat datagram sending and receiving threads of the backup scheduler are notified. Thus, the heartbeats of both the backup and primary schedulers return to normal.

[0154] After steps 3301-3302 provided in this embodiment, the determination of the node's working status can be completed.

[0155] The fault status of a node can be obtained based on its type in the following ways.

[0156] Step 3401: When the node is a standby scheduler or executor, obtain the current master scheduler, notify the master scheduler whose heartbeat thread is normal, and restore the normal heartbeat between the node and the master scheduler. When the node is a standby scheduler or executor, obtain the current master scheduler and restore the heartbeat datagram between the node and the master scheduler.

[0157] When the primary scheduler fails, the standby scheduler or executor may receive heartbeat datagrams from a non-primary scheduler. In order for the standby scheduler or executor to distinguish whether the received heartbeat datagram was sent by the primary scheduler, the fault detection thread needs to identify the correct primary scheduler, and the heartbeat datagram is recovered from the obtained primary scheduler.

[0158] Step 3402: When the node is the primary scheduler, determine whether the primary scheduler needs to switch to the backup scheduler, and switch according to the determination result, thereby restoring the normal heartbeat between the primary scheduler and the non-primary scheduler.

[0159] If the primary scheduler does not receive any heartbeat data packets, it needs to notify its fault detection thread to perform network fault detection. Upon receiving the notification, the primary scheduler's fault detection thread first accesses the metadata database. If the access fails, a network fault exists, and an alarm is issued. If no network fault exists, it determines whether the primary scheduler needs to be switched to the backup scheduler.

[0160] After steps 3401-3402 provided in this embodiment, the fault status can be obtained, and corresponding fault handling can be performed according to the fault status.

[0161] The master scheduler can be obtained in step 3401 using the following method.

[0162] Step 3501: The node's apparent death fault detection thread accesses the service table in the metadata, obtains the information of the main scheduler, and accesses the main scheduler through remote service call according to the obtained information.

[0163] The executor or standby scheduler accesses the SERVICE table in the metadata to obtain the main scheduler information and accesses the main scheduler via remote service calls.

[0164] Step 3502: If access is successful and the master scheduler is functioning correctly, set the master scheduler as the current scheduler and notify the heartbeat thread.

[0165] If the access is successful and it is indeed the primary scheduler, return this primary scheduler.

[0166] Step 3503: If normal access is not possible, put the fault detection thread into sleep according to the specified sleep cycle, and try to access the main scheduler again at the end of each sleep cycle until normal access is possible.

[0167] If a normal master scheduler cannot be obtained, the system will sleep for a period of time, such as 1 second, and continue to try until a normal master scheduler is obtained.

[0168] After steps 3501-3503 provided in this embodiment, the correct master scheduler can be obtained.

[0169] Furthermore, when a node is a standby scheduler, it may have already been switched to the primary scheduler. Therefore, obtaining the current primary scheduler requires the following steps.

[0170] Step 3601: When the current master scheduler is the node itself, determine whether the node has been set as the master scheduler.

[0171] Unlike the executor network failure detection thread which obtains the current primary scheduler, this backup scheduler may have already switched to the primary scheduler.

[0172] Step 3602: If it has already been set as the master scheduler, set the current master scheduler to the node itself and terminate the deadlock fault detection thread on the node.

[0173] The master scheduler obtained is itself, at which point the backup scheduler network fault detection thread needs to exit and terminate.

[0174] After steps 3601-3602 provided in this embodiment, the standby scheduler can obtain the correct primary scheduler.

[0175] In practice, when the primary scheduler fails, the following steps can be used to determine whether the primary scheduler needs to be switched to the backup scheduler, and the switch can be performed based on the determination result.

[0176] Step 3701: Obtain node information from the service table and determine whether the node type is a standby scheduler.

[0177] Retrieve your own scheduler information from the SERVICE table in the metadata database.

[0178] Step 3702: When the node type is a standby scheduler, switch the node to the standby scheduler, stop sending heartbeat data packets, and start the standby scheduler's dead fault detection thread.

[0179] If the scheduler type in the message has been changed to standby scheduler, it means that a primary scheduler already exists in the system, and the old primary scheduler needs to be switched to the standby scheduler. Stop the datagram sending and receiving threads and the network fault detection thread, stop the flow scheduling function, start the datagram sending and receiving threads of other schedulers and the network fault detection thread of the standby scheduler, and enable the standby scheduler function.

[0180] Step 3703: When the node type is the primary scheduler, do not switch the node type.

[0181] Otherwise, the current master scheduler will remain the master scheduler in the system without switching.

[0182] After steps 3701-3703 provided in this embodiment, the switchover of the faulty master scheduler can be completed.

[0183] Furthermore, during the master scheduler's revival process, it is unable to process the executor's event notifications. Therefore, when the master scheduler reconstructs the running instance, the running status notifications sent by the executor will wait until the master scheduler completes the corresponding process to prevent concurrent conflicts in the process running status settings.

[0184] Example 4:

[0185] Based on the distributed ETL master scheduler contention methods provided in Embodiments 1 to 3 above, the present invention also provides an apparatus for implementing the distributed ETL master scheduler contention methods described above, such as... Figure 6 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The device for distributed ETL master scheduler contention in this embodiment includes one or more processors 11 and a memory 12. Figure 6 Take a processor 11 as an example.

[0186] Processor 11 and memory 12 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0187] The memory 12 serves as a non-volatile computer-readable storage medium for a distributed ETL master scheduler contention method. It can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the distributed ETL master scheduler contention methods in Embodiments 1 to 3. The processor 11 executes various functional applications and data processing of the distributed ETL master scheduler contention apparatus by running the non-volatile software programs, instructions, and modules stored in the memory 12, thereby implementing the distributed ETL master scheduler contention methods in Embodiments 1 to 3.

[0188] Memory 12 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 12 may optionally include memory remotely located relative to processor 11, which can be connected to processor 11 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0189] The program instructions / modules are stored in memory 12. When executed by one or more processors 11, they perform the distributed ETL master scheduler contention method described in Embodiments 1 to 3 above, for example, performing the above-described... Figures 1 to 5 The steps shown.

[0190] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0191] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for contention in a distributed ETL master scheduler, characterized in that, include: When the current scheduler starts up, and / or when the current scheduler attempts to switch to the primary scheduler, the current scheduler creates a primary scheduler contention table in the metadata database, using the primary scheduler contention table as a distributed mutex lock between different schedulers. After the current scheduler successfully creates the master scheduler contention table and acquires the distributed mutex lock, it determines whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and the existing master scheduler information stored in the metadata database. The information of the current scheduler includes ID, host address and port number. If the ID of the current scheduler is the same as the ID of the existing master scheduler, the host address of the current scheduler is the same as the host address of the existing master scheduler, and the port number of the current scheduler is the same as the port number of the existing master scheduler, the current scheduler is set as the master scheduler and the information of the master scheduler is updated to the information of the current scheduler. If the host address of the current scheduler is different from the host address of the existing master scheduler, and / or the port number of the current scheduler is different from the port number of the existing master scheduler, determine the running status of the existing master scheduler and configure the master scheduler according to the running status; The process of determining the operating status of the existing master scheduler and setting it accordingly includes: sending at least one network response request to the existing master scheduler to determine if it can respond normally; if it cannot respond normally, setting the current scheduler as the master scheduler and updating its information; if it can respond normally and its ID is the same as the existing master scheduler's ID, modifying its ID and re-competing for master scheduler status; if it can respond normally and its ID is different from the existing master scheduler's ID, setting it as the backup scheduler and not updating the existing master scheduler's information.

2. The method for contention in a distributed ETL master scheduler according to claim 1, characterized in that, The current scheduler creates a master scheduler contention table in the metadata database, specifically including: When the current scheduler successfully requests to create the master scheduler contention table and acquires the distributed mutex lock, if there is no master scheduler in the scheduler information stored in the metadata database, the current scheduler becomes the master scheduler and updates the scheduler information stored in the metadata database.

3. The method for contention in a distributed ETL master scheduler according to claim 1, characterized in that, The current scheduler creates a master scheduler contention table in the metadata database, which also includes: If the current scheduler fails to create the master scheduler contention table, it will put the current scheduler into sleep mode according to the specified sleep period, and try to create the master scheduler contention table again at the end of each sleep period, until the master scheduler contention table is successfully created and the distributed mutex lock is acquired.

4. The method for contention in a distributed ETL master scheduler according to claim 1, characterized in that, The phrase "when the current scheduler starts" also includes: After the current scheduler successfully acquires a distributed lock, it starts listening for services at the specified host address and port number so that other nodes can access the current scheduler.

5. The method for contention in a distributed ETL master scheduler according to claim 1, characterized in that, After determining whether the current scheduler can be set as the master scheduler based on the current scheduler's information and the existing master scheduler information stored in the metadata database, the process further includes: Delete the scheduling contention table created by the current scheduler to release the distributed mutex lock.

6. The method for contention in a distributed ETL master scheduler according to claim 1, characterized in that, The metadata database also includes a service table. After determining whether the current scheduler can be set as the master scheduler based on the information of the current scheduler and the existing master scheduler information stored in the metadata database, the database further includes: When the main scheduler starts, it inserts or updates its own information in the service table of the metadata database; When a non-master scheduler starts up, it registers itself with the master scheduler. The master scheduler then inserts or updates the information of the corresponding node in the service table based on the information registered by the node. The node starts a fake death fault detection thread. Based on the information of the master scheduler and non-master scheduler recorded in the receiving service table, the fake death fault detection thread receives the heartbeat abnormal notification of each node and restores the normal heartbeat of the node with network failure.

7. The method for contention in a distributed ETL master scheduler according to claim 1, characterized in that, After determining whether the current scheduler can be set as the master scheduler based on the current scheduler's information and the existing master scheduler information stored in the metadata database, the process further includes: The master scheduler schedules the process to the corresponding executor. When the process starts or ends, the executor notifies the master scheduler of the process's running status. The main scheduler caches the corresponding running instances based on the received running status, generates corresponding running logs in the metadata database, uses the running logs to cache information for each process, as well as the parent-child relationship between different processes, and obtains monitoring information consistent with the running status of the processes. When the main scheduler detects an executor failure, it processes the process in the executor through the running instance cached by the main scheduler, so that the running status of the processed process is consistent with the monitoring information. When the master scheduler restarts or switches, all running process instances in all executors are revived through the process execution log, so that all process execution instances managed by the new master scheduler are completely consistent with the master scheduler before the failure.

8. A device for contention among distributed ETL master schedulers, characterized in that: The system includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. When executed by the processor, the instructions are used to perform the method of contention for the distributed ETL master scheduler as described in any one of claims 1-7.

Images