An ETL system and a data migration method of a distributed cluster architecture

Abstract

The application relates to the technical field of ETL, and provides an ETL system of a distributed cluster architecture and a data migration method.The ETL system of the distributed cluster architecture comprises a manager, at least one scheduler and at least one executor, the manager, the scheduler and the executor are arranged as independent function modules, the manager is used for receiving an external request, configuring a data synchronization process according to the external request, and receiving data from the scheduler; the scheduler is used for calculating a trigger condition of the data synchronization process, and delivering the data synchronization process to a corresponding executor, and receiving information from the executor; and the executor is used for executing the data synchronization process, and sending running state information of the data synchronization process to the scheduler. Since the manager, the scheduler and the executor are independent function modules, the manager, the scheduler and the executor are suitable for dynamic deployment of a cloud service system, and are suitable for adapting to a task of synchronizing large-scale data sources located at different physical positions to a central database during data synchronization application deployment.

Description

Technical Field

[0001] This invention relates to the field of ETL technology, and in particular to an ETL system with a distributed cluster architecture and a data migration method. Background Technology

[0002] In ETL (Extraction-Transformation-Loading) systems, data synchronization tasks are typically represented as a data synchronization process. This process usually includes information such as the source data source, the destination data source, and data transformation rules. Based on this information, the ETL system migrates data from one data source to another, and the data may be transformed during the migration process.

[0003] The basic function of ETL is to use a network to synchronize data from one data source to another. If the two data sources are physically far apart, then the network is the Internet. However, for security reasons, the data source service ports are generally not directly exposed to the Internet. Therefore, a compromise is adopted, adding a "relay source" and configuring multiple ETL servers to synchronize the data from three data sources in different physical locations to the central database in the central location. This is the distributed deployment of ETL.

[0004] Centralized ETL architectures are unsuitable for large-scale synchronization of data from data sources located in different physical locations to a central database. In traditional distributed ETL deployments, each data source requires a complete centralized ETL system, including a metadata database to support its operation. Metadata required by each ETL system is stored independently, but the dispersed storage of metadata across multiple databases increases the risk of data leakage. Secondly, there is a lack of a universal management and monitoring platform for the entire data synchronization application system. Centralized ETL system clients manage and monitor individual ETL systems, while distributed data synchronization applications involve multiple ETL systems. This necessitates the custom development of a separate client management and monitoring system for the distributed data synchronization application. These two factors increase the complexity, workload, and implementation costs of deploying a centralized ETL system as a distributed data synchronization application, as well as the security risk of metadata leakage.

[0005] Furthermore, traditional centralized ETL architectures are unsuitable for the dynamic deployment of cloud service systems. One characteristic of cloud service systems is the on-demand allocation of computing resources. When deploying an ETL server in the cloud, cloud computing resources are dynamically requested based on the number and volume of data synchronization tasks. One approach is for the ETL server to automatically request or release computing resources from the cloud service system on demand; another approach is to dynamically add or remove data synchronization executors on the cloud service system based on demand. If cloud computing resources are requested using the former approach, resource redundancy will occur. Because the scheduler, manager, and executors in a traditional centralized ETL architecture are centralized, it is impossible to dynamically add or remove executors independently.

[0006] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide an ETL system and data migration method with a distributed cluster architecture. This invention addresses the problem that when synchronizing data from data sources located in different physical locations to a central database using a traditional centralized ETL architecture, the centralized ETL system needs to be deployed as a distributed data synchronization application. This deployment method is complex, labor-intensive, and has high implementation and operation costs. It also addresses the problem that when a traditional centralized ETL architecture is used for the dynamic deployment of cloud service systems, it is impossible to dynamically add or remove executors.

[0008] In a first aspect, the present invention provides a distributed cluster architecture ETL system, comprising: a manager, at least one scheduler, and at least one executor, wherein:

[0009] The manager is used to receive external requests, configure the data synchronization process according to the external requests, provide registration services to the scheduler, and receive data from the scheduler.

[0010] The scheduler is used to calculate the triggering conditions for each data synchronization process, and to allocate the data synchronization processes that meet the triggering conditions to the corresponding executors. It is also used to receive running status information from the executors and send the running status information to the manager.

[0011] The executor is used to execute the data synchronization process. During the operation of the data synchronization process, it acquires the running status information of the data synchronization process and sends the running status information to the scheduler.

[0012] Furthermore, when the ETL system includes multiple schedulers:

[0013] The manager is used to receive the registration request of the scheduler. During the initial registration, the master scheduler and the backup scheduler are determined according to the order of successful registration, or the master scheduler is determined by leader election or consensus algorithm.

[0014] The main scheduler is used to calculate the triggering conditions in real time and submit the data synchronization process that meets the triggering conditions to the designated executor. At the same time, the main scheduler collects the status information during the operation of the data synchronization process. After receiving the operation status information returned by the executor, the main scheduler synchronizes the operation status information to the backup scheduler.

[0015] Furthermore, when there are two schedulers, the schedulers are divided into a primary scheduler and a backup scheduler, and the ETL system adopts a primary-backup scheduler architecture.

[0016] When there are two or more schedulers, the schedulers are divided into one master scheduler and multiple backup schedulers, and the ETL system is a scheduler cluster architecture.

[0017] Furthermore, when a scheduler crashes and restarts, and then re-registers, if there is currently a normally functioning master scheduler, then all schedulers that re-register and join are backup schedulers.

[0018] When an executor fails to return information to the master scheduler, a new master scheduler is automatically generated from the backup schedulers, and the executor returns information to the new master scheduler.

[0019] The primary scheduler and the backup scheduler perform heartbeat detection on each other. Whenever a failure event is detected, an alarm is sent. If the primary scheduler fails, a new primary scheduler is automatically generated from the backup schedulers.

[0020] Among them, a new master scheduler is generated from the backup schedulers through leader election or consensus algorithm.

[0021] Furthermore, when an actuator fails to return information to the master scheduler, the ETL system also includes:

[0022] The actuator sends a verification request to the backup scheduler in the network via broadcast.

[0023] After receiving the verification request via broadcast, each backup scheduler compares its most recent heartbeat detection time with that of the main scheduler.

[0024] If the time difference between the next heartbeat detection and the next heartbeat detection is less than a preset value, the corresponding backup scheduler will initiate the heartbeat detection in advance; based on the heartbeat detection result, it is confirmed that a new master scheduler will be automatically generated from the backup scheduler, or it is confirmed that there is a problem with the network between the executor and the master scheduler.

[0025] The master scheduler sets a heartbeat detection time interval for each backup scheduler in the network and informs the backup schedulers under its control of the time interval.

[0026] Furthermore, each scheduler is configured with a metadata database, or all schedulers share a single metadata database;

[0027] The manager is directly connected to the metadata database, or the manager is directly connected to the metadata database through the scheduler;

[0028] When the manager is directly connected to the metadata database, the manager is used to interact with the metadata database and provide services to the outside world;

[0029] When the manager connects to the metadata database through the scheduler, the manager interacts with the metadata database through the scheduler to provide services to the outside world.

[0030] Furthermore, the scheduler is used to collect the running status information generated by the executor and append the running status information to a temporary log file in real time;

[0031] The scheduler is also used to add the running status information to the log information list so as to update the log information list in real time;

[0032] After the running status information is written to the metadata database, the corresponding running status information is removed from the log information list, and a savepoint marker is written to the temporary log file.

[0033] Furthermore, the executor maintains an append-only log file;

[0034] The executor is used to write the running status information of the data synchronization process into the append log file and send the running status information to the scheduler;

[0035] After the running status information is successfully sent, the executor is used to write a savepoint marker into the append log file;

[0036] After the executor crashes and restarts, the executor is used to restore the running state of the data synchronization process based on the appended log file.

[0037] Furthermore, when deploying data synchronization applications, the ETL system includes:

[0038] At least one executor is deployed on the data source side, and the manager and the scheduler are deployed in the same physical location on the central database side. At the same time, at least one executor is deployed on the central database side to complete the deployment of the ETL system.

[0039] The executor on the data source side is used to extract data from the data source and load it into the intermediate source, while the executor on the central database side is used to transform the data in the intermediate source and transfer the transformed data to the central database.

[0040] Secondly, the present invention also provides a data migration method for a distributed cluster architecture, wherein the data migration method is applied to an ETL system as described in the first aspect, the ETL system comprising: a manager, at least one scheduler, and at least one executor, wherein the manager, scheduler, and executor are all independent functional modules to provide independent services respectively, and the data migration method comprises:

[0041] The manager receives external requests, configures the data synchronization process according to the external requests, provides registration services to the scheduler, and receives data from the scheduler.

[0042] The scheduler calculates the triggering conditions for each data synchronization process, assigns the data synchronization processes that meet the triggering conditions to the corresponding executors, and also receives running status information from the executors and sends the running status information to the manager.

[0043] The executor executes the data synchronization process, and during the operation of the data synchronization process, it acquires the running status information of the data synchronization process and sends the running status information to the scheduler.

[0044] In this embodiment of the invention, the manager, scheduler, and executor are set as independent functional modules, working in coordination. The manager, at least one scheduler, and at least one executor can be set in different locations individually, or can be added or removed individually. The distributed cluster architecture ETL system can dynamically add or remove executors according to the number of data synchronization tasks and the amount of data, ensuring the matching of computing resources with the amount of data, avoiding resource redundancy, and is suitable for the dynamic deployment of cloud service systems. At the same time, when deploying data synchronization applications, the distributed cluster architecture ETL system does not need to deploy a complete ETL system on the data source side. It only needs to deploy a corresponding number of executors on the data source side, and then deploy the manager, scheduler, and a corresponding number of executors on the central database side. This reduces the complexity of the system, reduces the workload of deployment, reduces operating costs, and can adapt to large-scale tasks of synchronizing data from data sources located in different physical locations to the central database.

[0045] Furthermore, in this embodiment of the invention, the scheduler needs to register with the manager when it starts up, and the executor needs to register with the scheduler when it starts up. The executor sends the data synchronization process running status information to the scheduler, and the scheduler sends the running status information to the manager. Both the scheduler and the executor maintain logs, so that in the ETL system of the distributed cluster architecture, the manager, scheduler and executor have a complete management and monitoring system. The manager can manage and monitor the entire ETL system of the distributed cluster architecture. Therefore, the client can manage and monitor the entire ETL system through the manager.

[0046] Furthermore, in this embodiment of the invention, the metadata database is centralized on the scheduler side. When the distributed cluster architecture ETL system performs data synchronization application deployment, the metadata database does not need to be deployed at multiple data sources, thereby reducing the risk of metadata information leakage. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0048] Figure 1 This is a schematic diagram of the architecture of a distributed cluster architecture ETL system provided in an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the scheduler master-slave architecture of an ETL system with a distributed cluster architecture provided in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the scheduler cluster architecture of an ETL system with a distributed cluster architecture provided in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the independent metadata database architecture of a distributed cluster architecture ETL system provided in an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram illustrating the data synchronization application deployment of an ETL system with a distributed cluster architecture provided in an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of a distributed data synchronization application deployment of a traditional centralized ETL system provided by an embodiment of the present invention;

[0054] Figure 7This is a flowchart illustrating a data migration method for a distributed cluster architecture provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0056] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0057] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0058] Example 1:

[0059] Embodiment 1 of the present invention provides an ETL system with a distributed cluster architecture, such as Figure 1 As shown, it includes: a manager, at least one scheduler, and at least one executor, wherein:

[0060] The manager is used to receive external requests, configure the data synchronization process according to the external requests, provide registration services to the scheduler, and receive data from the scheduler.

[0061] The scheduler is used to calculate the triggering conditions for each data synchronization process, and to allocate the data synchronization processes that meet the triggering conditions to the corresponding executors. It is also used to receive running status information from the executors and send the running status information to the manager.

[0062] The executor is used to execute the data synchronization process. During the operation of the data synchronization process, it acquires the running status information of the data synchronization process and sends the running status information to the scheduler.

[0063] In this embodiment, the main functions of the manager, scheduler, and executor are as follows:

[0064] Manager: The user server of the ETL system, providing user operation and monitoring functions such as designing and managing the data synchronization process, monitoring process execution, managing data sources and monitoring status, and monitoring the system.

[0065] Scheduler: If the data synchronization process is set with triggering conditions such as timed triggering and event triggering, the scheduler monitors the triggering conditions in real time. If the triggering conditions are met, the scheduler submits the synchronization process to the executor for execution. At the same time, it realizes the load balancing of the executor, stores the process execution log, and forwards the monitoring information of the execution process.

[0066] Executor: Executes the data synchronization process submitted by the scheduler and returns monitoring information and execution logs during the process.

[0067] Specifically, the manager provides external user interface management services and API (Application Programming Interface) services to receive external requests. It also provides services such as a data synchronization process designer, data source management, system operation monitoring, server status monitoring, and data source status monitoring to implement Web (World Wide Web) management and monitoring services; and allows for the custom development of ETL applications through the API interface services.

[0068] When the scheduler assigns the data synchronization process to the executor, the scheduler determines whether the data synchronization process can be executed on multiple executors. If the data synchronization process can be executed on multiple executors, the scheduler considers load balancing among the executors when submitting the data synchronization process.

[0069] In this embodiment of the invention, the manager, scheduler, and executor are set as independent functional modules, working in coordination. The manager, at least one scheduler, and at least one executor can be set in different locations individually, or can be added or removed individually. The distributed cluster architecture ETL system can dynamically add or remove executors according to the number of data synchronization tasks and the amount of data, ensuring the matching of computing resources with the amount of data, avoiding resource redundancy, and is suitable for the dynamic deployment of cloud service systems. At the same time, when deploying data synchronization applications, the distributed cluster architecture ETL system does not need to deploy a complete ETL system on the data source side. It only needs to deploy a corresponding number of executors on the data source side, and then deploy the manager, scheduler, and a corresponding number of executors on the central database side. This reduces the complexity of the system, reduces the workload of deployment, reduces operating costs, and can adapt to large-scale tasks of synchronizing data from data sources located in different physical locations to the central database.

[0070] To ensure system reliability, the ETL system includes multiple schedulers, one of which serves as the master scheduler and the others as backup schedulers. When the ETL system includes multiple schedulers, the manager is used to receive registration requests from the schedulers. During initial registration, the master scheduler and backup schedulers are determined according to the order of successful registration, or a leader election or consensus algorithm is used to determine the master scheduler.

[0071] The master scheduler is used to calculate the triggering conditions in real time and submit the data synchronization process that meets the triggering conditions to the designated executor. At the same time, the master scheduler collects the status information during the operation of the data synchronization process. After receiving the operation status information returned by the executor, the master scheduler synchronizes the operation status information to the backup scheduler.

[0072] After the main scheduler successfully sends the data synchronization process to the executor, it will generate a unique data synchronization process execution ID and other information for the entire system, and synchronize the data synchronization process execution ID and other information to the backup scheduler.

[0073] In an optional scheme, after the master scheduler submits a data synchronization process that meets the triggering conditions to the designated executor, it designates a backup scheduler as the receiving scheduler to receive the running status information returned by the executor. After receiving the running status information returned by the executor, the receiving scheduler synchronizes the running status information to other schedulers.

[0074] In this embodiment, when the scheduler registers with the manager, the scheduler performs the registration process through a registration sub-thread. If the registration fails, it needs to be repeated after a period of time.

[0075] When determining the master scheduler and backup schedulers based on the order of successful registration, the scheduler that registers first during the initial registration becomes the master scheduler. When using leader election or consensus algorithm to determine the master scheduler, leader election or consensus algorithm is generally performed among all schedulers before the initial registration or after all schedulers have completed registration to determine the master scheduler.

[0076] In actual operation, the scheduler may crash. After the scheduler restarts, it will re-register. When the scheduler crashes and restarts and re-registers, if there is a normally operating master scheduler, the schedulers that re-register and join are all backup schedulers.

[0077] To better manage the executors, the executors need to register with the scheduler when they start up. If the registration fails, they need to register repeatedly after a period of time until the registration is successful. When there are multiple schedulers, the executors register with the main scheduler.

[0078] To enhance system stability, one could consider increasing the number of schedulers, such as... Figure 2 and Figure 3 As shown, in this embodiment, based on the basic system architecture of the ETL system, two architecture methods for the ETL system are provided.

[0079] like Figure 2 As shown, when there are two schedulers, the schedulers are divided into a primary scheduler and a backup scheduler, and the ETL system adopts a primary-backup scheduler architecture.

[0080] like Figure 3 As shown, when there are two or more schedulers, the schedulers are divided into one master scheduler and multiple backup schedulers, and the ETL system is a scheduler cluster architecture.

[0081] The central component of a distributed cluster architecture ETL system is the scheduler. If the scheduler fails, the entire system will be paralyzed. Compared with an ETL architecture using a single scheduler, the master-slave architecture ETL system and the cluster architecture ETL system have multiple schedulers, and their stability is significantly better than that of a single scheduler ETL system.

[0082] At this time, the manager can accept data synchronization process running status information submitted by multiple schedulers, and push the running status information to the client to realize the running status monitoring of the data synchronization process on the client.

[0083] To ensure that a primary scheduler is always available to respond to requests from executors and managers, the primary scheduler is not fixed in practice and will switch between primary and backup schedulers based on actual circumstances. In this embodiment, the backup scheduler will automatically become the primary scheduler in some situations, such as:

[0084] When an executor fails to return information to the master scheduler, a new master scheduler is automatically generated from the backup schedulers, and the executor returns information to the new master scheduler.

[0085] The primary scheduler and the backup scheduler perform heartbeat detection on each other. Whenever a failure event is detected, an alarm is sent. If the primary scheduler fails, a new primary scheduler is automatically generated from the backup schedulers.

[0086] When the manager sends a command to the master scheduler to start the data synchronization process, if the submission fails or the master scheduler is detected to be down, a new master scheduler is automatically generated in the backup scheduler, and the manager sends a command to the new master scheduler.

[0087] The process of automatically generating a master scheduler in the backup scheduler includes:

[0088] When the total number of schedulers is two, that is, when there was originally only one master scheduler and one backup scheduler, the backup scheduler automatically becomes the new master scheduler, and the original master scheduler becomes the backup scheduler after returning to normal.

[0089] When the total number of schedulers is greater than two, that is, when there is originally one master scheduler and multiple backup schedulers, a leader election is conducted among the backup schedulers to generate a new master scheduler. After the original master scheduler returns to normal, it can become a backup scheduler or replace the newly generated master scheduler in the previous round of leader election, depending on the specific leader election algorithm, and directly become the master scheduler.

[0090] To determine the reason why the actuator failed to return information to the master scheduler, the ETL system further includes the following when the actuator fails to return information to the master scheduler:

[0091] The actuator sends a verification request to the backup scheduler in the network via broadcast.

[0092] After receiving the verification request via broadcast, each standby scheduler compares its most recent heartbeat detection time with that of the main scheduler.

[0093] If the time difference between the next heartbeat detection and the next heartbeat detection is less than a preset value, the corresponding backup scheduler will initiate a heartbeat detection in advance. Based on the heartbeat detection result, it is confirmed that a new master scheduler will be automatically generated from the backup scheduler, or it is confirmed that there is a problem with the network between the executor and the master scheduler.

[0094] The master scheduler sets a heartbeat detection time interval for each backup scheduler in the network and informs the backup schedulers under its control of the time interval.

[0095] Considering that in practical application scenarios, when the primary scheduler and the backup schedulers perform heartbeat detection, data congestion may occur if the primary scheduler processes heartbeat detections initiated by multiple backup schedulers at the same time. Therefore, the primary scheduler sets a heartbeat detection timetable, which specifies the time interval for each scheduler to initiate heartbeat detection, and distributes it to all backup schedulers. Each backup scheduler sends heartbeat detections to the primary scheduler at preset time intervals according to the heartbeat detection timetable. If the primary scheduler's heartbeat stops, i.e., the primary scheduler crashes, the backup scheduler that detects the primary scheduler's crash initiates a leader election to the remaining backup schedulers in the network via broadcast, resulting in a new primary scheduler.

[0096] When the executor fails to return information to the primary scheduler, the executor broadcasts a verification request to the backup schedulers in the network. The backup schedulers check the heartbeat check time table and initiate the heartbeat check in advance from the backup scheduler that needs to initiate a heartbeat check to the primary scheduler. This determines whether the problem is with the network between the executor and the primary scheduler or with the primary scheduler itself. If the problem is with the primary scheduler, the backup scheduler that discovered the problem initiates a leader election to generate a new primary scheduler.

[0097] To better manage the data and improve the interaction between the manager and the metadata database, such as... Figures 1-4 As shown, each scheduler is configured with one metadata database, or all schedulers share one metadata database;

[0098] The manager is directly connected to the metadata database, or the manager is directly connected to the metadata database through the scheduler;

[0099] When the manager is directly connected to the metadata database, the manager is used to interact with the metadata database and provide services to the outside world;

[0100] When the manager connects to the metadata database through the scheduler, the manager interacts with the metadata database through the scheduler to provide services to the outside world.

[0101] like Figure 2 and Figure 3 As shown, in the ETL system with a master-slave scheduler architecture and the ETL system with a cluster architecture, each scheduler is configured with a metadata database. Whenever the master scheduler or the associated metadata database fails, the ETL system can quickly switch a backup scheduler to become the master scheduler, and then access the metadata database of the new master scheduler.

[0102] Among them, such as Figure 4 As shown, when all the schedulers share a single metadata database, the ETL system is a distributed ETL architecture with independent metadata databases. That is, in the master-slave scheduler architecture and the cluster architecture ETL system, the metadata databases of all schedulers are separated, and all schedulers share a single metadata database. The manager can directly connect to the metadata database (e.g., ...). Figure 4 As shown), the manager can also connect to the metadata database through the scheduler.

[0103] The distributed ETL architecture of the independent metadata database simplifies the process of the scheduler storing information in the metadata database; and in the actual deployment of the distributed ETL architecture of the independent metadata database, the scheduler, manager and metadata database are generally located in the same local area network, and the manager is directly connected to the metadata database.

[0104] like Figures 1-4 As shown, this embodiment provides two methods for interaction between the manager and the metadata database. One method involves the manager directly connecting to the metadata database. When the manager is directly connected to the metadata database, it interacts with the metadata database and provides services externally.

[0105] Another approach is that the manager connects to the metadata through the scheduler. When the manager connects to the metadata database through the scheduler, the manager interacts with the metadata database through the scheduler to provide services to the outside world.

[0106] Specifically, when the manager is directly connected to the metadata database, it can interact with the metadata database directly or indirectly through the scheduler. In this case, the manager can function as an independent server, bypassing the scheduler to directly provide services such as data synchronization process design, data synchronization process management, and data source management. Otherwise, it needs to interact with the metadata database through the scheduler to provide normal external services.

[0107] To manage and monitor the ETL system, particularly the data synchronization process, in this embodiment, the scheduler and executor are also used for log maintenance. Specifically:

[0108] The scheduler collects runtime status information generated by the executors and appends this information to a temporary log file in real time. The scheduler also adds the runtime status information to a log information list to update the list in real time. When the data synchronization process finishes, the complete runtime status information is written to the metadata database.

[0109] After the running status information is written to the metadata database, the corresponding running status information is removed from the log information list, and a savepoint marker is written to the temporary log file.

[0110] When multiple schedulers are available, the primary scheduler selects a backup scheduler for log maintenance. When a new primary scheduler is created, it initiates data synchronization, synchronizing the log information lists and temporary log files between the new primary scheduler and the backup scheduler. Services are only provided to the manager after the states are consistent. After a scheduler crashes and restarts, it needs to synchronize and restore the running state information of the current data synchronization process from the temporary log files and the current primary scheduler.

[0111] After receiving the running status information of the data synchronization process returned by the executor, the scheduler needs to send the running status information to the manager and append the running status information to the temporary log file. When the data synchronization process ends, the complete running status information is written to the metadata database to form a process running log. After the process running log is successfully saved, the scheduler writes a savepoint marker to the temporary log file, and the schedulers synchronize the process running logs with each other.

[0112] The executor maintains an append log file. The executor is used to write the running status information of the data synchronization process into the append log file and send the running status information to the scheduler. After the running status information is successfully sent, the executor is used to write a savepoint marker into the append log file. After the executor crashes and restarts, the executor is used to restore the running status of the data synchronization process according to the append log file.

[0113] To manage and monitor the data source, in this embodiment, the executor is also used to collect data source status information. Specifically, the scheduler submits a request to collect data source information to the executor. After receiving the request, the scheduler collects the data source status information and sends it to the executor. The scheduler then sends the data source status information to the manager. The manager, after obtaining the data source status information, pushes it to the client, thus enabling data source status monitoring.

[0114] In the process of acquiring data source status information, if the executor can connect to the specified data source, the executor will repeatedly collect the data source status information of the specified data source at certain time intervals and send the data source status information to the scheduler.

[0115] When multiple schedulers are present, the scheduler designated to collect data source status information will synchronize the data source status information to other schedulers after obtaining it. The manager can also accept data source status information submitted by multiple schedulers.

[0116] In this embodiment of the invention, the scheduler needs to register with the manager when it starts up, and the executor needs to register with the scheduler when it starts up. The executor sends the data synchronization process running status information to the scheduler, and the scheduler sends the running status information to the manager. Both the scheduler and the executor maintain logs, so that in the ETL system of the distributed cluster architecture, the manager, scheduler and executor have a complete management and monitoring system. The manager can manage and monitor the entire ETL system of the distributed cluster architecture. Therefore, the client can manage and monitor the entire ETL system through the manager.

[0117] Furthermore, in this embodiment of the invention, the metadata database is centralized on the scheduler side. When the distributed cluster architecture ETL system performs data synchronization application deployment, the metadata database does not need to be deployed at multiple data sources, thereby reducing the risk of metadata information leakage.

[0118] Example 2:

[0119] This embodiment provides a data synchronization application deployment example of an ETL system based on the distributed cluster architecture described in Embodiment 1, such as... Figure 5 As shown, the deployment includes: deploying at least one executor on the data source side, with the manager and the scheduler deployed at the same physical location on the central database side, and at least one executor also deployed on the central database side, thus completing the deployment of the ETL system.

[0120] The executor on the data source side is used to extract data from the data source and load it into the intermediate source; the executor on the central database side is used to transform the data in the intermediate source and transfer the transformed data to the central database.

[0121] At this point, the manager configures the data synchronization process based on the external request, the scheduler assigns the data synchronization process to the corresponding executor, the executor is responsible for executing the data synchronization process, and the executor sends the data synchronization process running status information to the scheduler, which then sends the running status information back to the manager.

[0122] Specifically, the executor deployed at the data source is on the same local area network as the data source. The executor on the data source side can access the data source and extract data from it. The data source-side executor then loads the extracted data into a relay source via the internet. Furthermore, the executor on the central database side is on the same local area network as the central database. The central database-side executor extracts data from the relay source and, according to the data synchronization process rules, performs or does not perform data conversion before loading the data into the central database, completing one data synchronization cycle. Both the data source-side and central database-side executors operate under the coordination of the scheduler.

[0123] The manager, scheduler, and metadata database are located on the same local area network (LAN). For data security reasons, the metadata database server is not exposed outside the LAN. The executor cannot directly access the metadata database; it must interact indirectly with the metadata database through the scheduler. The manager and metadata database are on the same LAN; the manager can interact directly with the metadata database or indirectly through the scheduler.

[0124] like Figure 6 As shown, when traditional centralized ETL is deployed in distributed data applications, each data source needs to deploy a complete centralized ETL system, and the central database also needs to deploy a complete ETL system. Each ETL system also needs to be configured with a metadata database.

[0125] Deploying multiple metadata databases in different locations increases the risk of data leakage. For example, multiple metadata databases need to be deployed at multiple data sources, and the risk of data leakage increases progressively with the geographical distribution of these databases. Furthermore, managing and monitoring multiple ETL systems is difficult, requiring the custom development of a management and monitoring platform for the entire data synchronization application system. These issues increase the complexity, workload, and implementation and operational costs of deploying a centralized ETL system as a distributed data synchronization application, as well as the security risk of metadata leakage. Therefore, the centralized ETL architecture is unsuitable for large-scale synchronization of data from data sources located in different physical locations to a central database.

[0126] When the ETL system in the distributed cluster architecture is deployed as a data synchronization application, the entire distributed cluster architecture ETL system can be managed and monitored through the manager, including the data source side executor and the central database side executor. At the same time, the metadata database is only deployed on the central database side, effectively reducing the risk of metadata leakage. This makes the distributed cluster architecture ETL system suitable for large-scale synchronization of data from data sources located in different physical locations to the central database.

[0127] Meanwhile, centralized ETL systems are not suitable for cloud deployment. When a centralized ETL system is deployed in the cloud, the number of deployments of the entire ETL system needs to be dynamically increased or decreased on the cloud service system according to demand, or the ETL server itself can automatically apply for or release computing resources from the cloud service system as needed.

[0128] However, when the ETL system with the distributed cluster architecture is deployed in a cloud environment, the number of executors can be dynamically increased or decreased according to the needs of the data synchronization process, which can maximize synchronization performance while minimizing cloud resource consumption costs.

[0129] Example 3:

[0130] Based on the aforementioned Embodiments 1 and 2, this embodiment also provides a data migration method for a distributed cluster architecture. This data migration method is applied to the ETL system of the distributed cluster architecture described in this embodiment. The ETL system includes: a manager, at least one scheduler, and at least one executor. The manager, scheduler, and executor are all independent functional modules, each providing independent services, such as... Figure 7 As shown, the data migration method includes:

[0131] Step 101: The manager receives external requests, configures the data synchronization process according to the external requests, provides registration services to the scheduler, and receives data from the scheduler.

[0132] The manager provides an external user interface management service and API interface service to receive external requests. It also offers services such as a data synchronization process designer, data source management, system operation monitoring, server status monitoring, and data source status monitoring to enable web management and monitoring services; and allows for the custom development of ETL applications through the API interface service.

[0133] Step 102: The scheduler calculates the triggering conditions for each data synchronization process, assigns the data synchronization processes that meet the triggering conditions to the corresponding executors, receives the running status information from the executors, and sends the running status information to the manager.

[0134] When the scheduler assigns the data synchronization process to the executor, the scheduler determines whether the data synchronization process can be executed on multiple executors. If the data synchronization process can be executed on multiple executors, the scheduler considers load balancing among the executors when submitting the data synchronization process.

[0135] Step 103: The executor executes the data synchronization process. During the operation of the data synchronization process, it obtains the running status information of the data synchronization process and sends the running status information to the scheduler.

[0136] In this embodiment, the manager can accept data synchronization process running status information submitted by the scheduler and push the running status information to the client to monitor the running status of the data synchronization process on the client. The manager can also accept data source status information submitted by multiple schedulers and push the data source information to the client to monitor the data source status on the client.

[0137] To enhance system stability, it is advisable to increase the number of schedulers and set up a master scheduler and a backup scheduler. The specific methods for setting up the master scheduler and backup scheduler are described above and will not be repeated here.

[0138] To better manage the data, in this embodiment, the ETL system also includes at least one metadata database. The metadata database can be distributed in at least two ways: each scheduler can configure one metadata database; or all schedulers can share one metadata database. Furthermore, the manager can connect directly to the metadata database, or the manager can connect to the metadata database through the scheduler. For details, please refer to the preceding description, which will not be repeated here.

[0139] For details on the deployment methods and working mechanisms of the manager, caller, and executor, please refer to the previous descriptions, which will not be repeated here.

[0140] In this embodiment of the invention, the manager, scheduler, and executor are set as independent functional modules. The manager, at least one scheduler, and at least one executor can be set in different locations or added or removed individually. The distributed cluster architecture ETL system can dynamically add or remove executors according to the number of data synchronization tasks and the amount of data, ensuring the matching of computing resources with the amount of data, avoiding resource redundancy, and is suitable for the dynamic deployment of cloud service systems. At the same time, when deploying data synchronization applications, the distributed cluster architecture ETL system does not need to deploy a complete ETL system on the data source side. It only needs to deploy a corresponding number of executors on the data source side, and then deploy the manager, scheduler, and a corresponding number of executors on the central database side. This reduces the complexity of the system, reduces the workload of deployment, reduces operating costs, and can adapt to large-scale tasks of synchronizing data from data sources located in different physical locations to the central database.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An ETL system with a distributed cluster architecture, characterized in that, include: The system comprises a manager, at least one scheduler, and at least one executor, wherein: The manager is used to receive external requests, configure the data synchronization process according to the external requests, provide registration services to the scheduler, and receive data from the scheduler. The scheduler is used to calculate the triggering conditions for each data synchronization process, and to allocate the data synchronization processes that meet the triggering conditions to the corresponding executors. It is also used to receive running status information from the executors and send the running status information to the manager. The executor is used to execute the data synchronization process. During the operation of the data synchronization process, it acquires the running status information of the data synchronization process and sends the running status information to the scheduler. The ETL system includes multiple schedulers, one of which acts as the master scheduler and the others as backup schedulers. The master scheduler sets up a heartbeat detection timetable, which specifies the time interval for each scheduler to initiate a heartbeat detection and sends it to all the backup schedulers. Each backup scheduler sends a heartbeat detection to the master scheduler at regular intervals according to the preset time interval in the heartbeat detection timetable. When the executor fails to return information to the primary scheduler, it broadcasts a verification request to the backup schedulers in the network. The backup schedulers check the heartbeat detection timetable and initiate a heartbeat detection early on from the backup scheduler that needs to initiate a heartbeat detection to the primary scheduler. This determines whether the problem lies in the network between the executor and the primary scheduler or with the primary scheduler itself. If the problem lies with the primary scheduler, the backup scheduler that detected the problem initiates a leader election to produce a new primary scheduler.

2. The ETL system with a distributed cluster architecture according to claim 1, characterized in that, When the ETL system includes multiple schedulers: The manager is used to receive the registration request of the scheduler. During the initial registration, the master scheduler and the backup scheduler are determined according to the order of successful registration, or the master scheduler is determined by leader election or consensus algorithm. The main scheduler is used to calculate the triggering conditions in real time and submit the data synchronization process that meets the triggering conditions to the designated executor. At the same time, the main scheduler collects the status information during the operation of the data synchronization process. After receiving the operation status information returned by the executor, the main scheduler synchronizes the operation status information to the backup scheduler.

3. The ETL system with a distributed cluster architecture according to claim 2, characterized in that: When there are two schedulers, the schedulers are divided into a primary scheduler and a backup scheduler, and the ETL system adopts a primary and backup scheduler architecture. When there are two or more schedulers, the schedulers are divided into one master scheduler and multiple backup schedulers, and the ETL system is a scheduler cluster architecture.

4. The ETL system with a distributed cluster architecture according to claim 2, characterized in that: When a scheduler crashes and restarts, and then re-registers, if there is a currently operating master scheduler, then any scheduler that re-registers and joins will be a backup scheduler. When an executor fails to return information to the master scheduler, a new master scheduler is automatically generated from the backup schedulers, and the executor returns information to the new master scheduler. The primary scheduler and the backup scheduler perform heartbeat detection on each other. Whenever a failure event is detected, an alarm is sent. If the primary scheduler fails, a new primary scheduler is automatically generated among the backup schedulers. Among them, a new master scheduler is generated from the backup schedulers through leader election or consensus algorithm.

5. The ETL system with a distributed cluster architecture according to claim 4, characterized in that, When an executor fails to return information to the master scheduler, the ETL system further includes: The actuator sends a verification request to the backup scheduler in the network via broadcast. After receiving the verification request via broadcast, each backup scheduler compares its most recent heartbeat detection time with that of the main scheduler. If the time difference between the next heartbeat detection and the next heartbeat detection is less than a preset value, the corresponding backup scheduler will initiate the heartbeat detection in advance; based on the heartbeat detection result, it is confirmed that a new master scheduler will be automatically generated from the backup scheduler, or it is confirmed that there is a problem with the network between the executor and the master scheduler. The master scheduler sets a heartbeat detection time interval for each backup scheduler in the network and informs the backup schedulers under its control of the time interval.

6. The ETL system with a distributed cluster architecture according to claim 1, characterized in that, Each scheduler is configured with a metadata database, or all schedulers share a single metadata database; The manager is directly connected to the metadata database, or the manager is directly connected to the metadata database through the scheduler; When the manager is directly connected to the metadata database, the manager is used to interact with the metadata database and provide services to the outside world; When the manager connects to the metadata database through the scheduler, the manager interacts with the metadata database through the scheduler to provide services to the outside world.

7. The ETL system with a distributed cluster architecture according to any one of claims 1 to 6, characterized in that: The scheduler is used to collect the running status information generated by the executor and append the running status information to the temporary log file in real time; The scheduler is also used to add the running status information to the log information list so as to update the log information list in real time; When the data synchronization process ends, the complete running status information is written into the metadata database; After the running status information is written to the metadata database, the corresponding running status information is removed from the log information list, and a savepoint marker is written to the temporary log file.

8. The ETL system with a distributed cluster architecture according to any one of claims 1 to 6, characterized in that, The executor maintains an append-only log file; The executor is used to write the running status information of the data synchronization process into the append log file and send the running status information to the scheduler; After the running status information is successfully sent, the executor is used to write a savepoint marker into the append log file; After the executor crashes and restarts, the executor is used to restore the running state of the data synchronization process based on the appended log file.

9. The ETL system with a distributed cluster architecture according to any one of claims 1 to 6, characterized in that, When deploying data synchronization applications in the ETL system of the distributed cluster architecture, the following are included: At least one executor is deployed on the data source side, and the manager and the scheduler are deployed in the same physical location on the central database side. At the same time, at least one executor is deployed on the central database side to complete the deployment of the ETL system. The executor on the data source side is used to extract data from the data source and load it into the intermediate source, while the executor on the central database side is used to transform the data in the intermediate source and transfer the transformed data to the central database.

10. A data migration method for a distributed cluster architecture, wherein the data migration method is applied to an ETL system with a distributed cluster architecture as described in any one of claims 1 to 9, characterized in that, The ETL system includes: a manager, at least one scheduler, and at least one executor, wherein the manager, scheduler, and executor are all independent functional modules to provide independent services. The data migration method includes: The manager receives external requests, configures the data synchronization process according to the external requests, provides registration services to the scheduler, and receives data from the scheduler. The scheduler calculates the triggering conditions for each data synchronization process, assigns the data synchronization processes that meet the triggering conditions to the corresponding executors, and also receives running status information from the executors and sends the running status information to the manager. The executor executes the data synchronization process, and during the operation of the data synchronization process, it acquires the running status information of the data synchronization process and sends the running status information to the scheduler.

Images