Scheduling migration method and device for under-cloud scheduling cluster, electronic equipment and server

Abstract

The invention provides an under-cloud scheduling cluster scheduling migration method and device, electronic equipment and a server, and relates to the technical field of data scheduling, and the scheme comprises the following steps: after an on-cloud standby scheduling cluster is created, a scheduling decorator is adopted to deploy and initialize the same job task as an under-cloud main scheduling cluster on the on-cloud standby scheduling cluster, and the job task starts to be executed; at the moment, if the on-cloud standby scheduling task fails to execute the job task due to factors such as environment and the like, the execution of the job task by the under-cloud main scheduling cluster is not influenced, and only when the on-cloud standby scheduling cluster and the under-cloud main scheduling cluster successfully execute the job task, the on-cloud standby scheduling cluster and the under-cloud main scheduling cluster are subjected to identity exchange; the on-cloud standby scheduling cluster is used as the main scheduling cluster, and the under-cloud main scheduling cluster is used as the standby scheduling cluster, so that the on-cloud standby scheduling cluster can successfully execute the scheduling task after the task is clouded.

Description

Technical Field

[0001] This invention relates to the field of data scheduling technology, specifically to a method, apparatus, electronic device, and server for migrating a cloud-based scheduling cluster. Background Technology

[0002] Because the scheduling cluster server currently used for ETL task scheduling and execution on the big data open platform is a physical server, expansion is cumbersome and hardware resources are limited. Due to hardware issues and cost considerations, there is an urgent need to move the scheduling cluster to the cloud.

[0003] The existing approach typically involves building a scheduling cluster in the cloud and then migrating the old scheduling cluster to the cloud cluster in batches. However, this approach of migrating tasks to the cloud in batches is prone to situations where tasks fail to run due to environmental or network issues after migration to the cloud. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, electronic device and server for migrating a cloud-based scheduling cluster, so as to ensure that the migrated scheduling cluster can operate reliably.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A method for migrating scheduling clusters in a cloud environment includes:

[0007] The scheduling decorator initializes the job tasks obtained from the on-premises primary scheduling cluster to the cloud-based backup scheduling cluster, and the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster have a primary-backup relationship.

[0008] Start the cloud-based backup scheduling cluster and determine the execution method corresponding to the node based on the node type of each process in the job task;

[0009] Determine whether the execution status of the job tasks in the cloud-based backup scheduling cluster is consistent with that in the on-premises primary scheduling cluster;

[0010] When the task execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster is consistent, the identities of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster are swapped.

[0011] Optionally, in the above-mentioned on-premises scheduling cluster migration method, determining the execution method corresponding to a node based on the node type of each process in the job task includes:

[0012] When the node type of the process in the job task is a data access node or a data processing / conversion node, the process will be executed independently in the cloud backup scheduling cluster and the on-premises main scheduling cluster.

[0013] When the node type of the process in the job task is a data output / service node, the process will be executed independently in the main scheduling cluster on the cloud, and the backup scheduling cluster on the cloud will not execute the process.

[0014] Optionally, in the above-mentioned on-premises scheduling cluster migration method, when the node type of the process in the job task is a data output / service node, it also includes:

[0015] The execution status of this process in the on-premises main scheduling cluster is obtained through the scheduling decorator;

[0016] When the execution status of the process in the primary scheduling cluster under the cloud is successful, the execution status of the task in the backup scheduling cluster on the cloud is marked as successful.

[0017] When the execution status of the process in the primary scheduling cluster on the cloud is marked as failed, the execution status of the task in the backup scheduling cluster on the cloud is marked as failed.

[0018] Optionally, in the above-mentioned on-premises scheduling cluster migration method, determining whether the job execution status of the on-premises backup scheduling cluster and the on-premises primary scheduling cluster is consistent includes:

[0019] When the working time of the cloud-based backup scheduling cluster reaches the first calibration time or the amount of jobs processed reaches the target amount, the execution status of each job in the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster is the same, indicating that the job execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster is consistent. Otherwise, it indicates that the job execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster is inconsistent.

[0020] Optionally, in the above-mentioned on-premises scheduling cluster migration method, after exchanging identities between the on-premises backup scheduling cluster and the on-premises primary scheduling cluster, the method further includes:

[0021] Determine whether the cloud-based backup scheduling cluster encounters any abnormalities while executing job tasks within the second calibration period. If an abnormality occurs, swap the identities of the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster again.

[0022] If no abnormality occurs, and the execution results of the job tasks are the same when the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster execute the job tasks, then the backup scheduling cluster is shut down.

[0023] Optionally, in the above-mentioned on-premises scheduling cluster migration method, after determining whether the job execution status of the on-premises backup scheduling cluster and the on-premises primary scheduling cluster is consistent, the method includes:

[0024] The tasks whose execution status is inconsistent between the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster are marked, and the execution log of the cloud-based backup scheduling cluster when executing the task is recorded.

[0025] Optionally, in the above-mentioned on-premises scheduling cluster migration method, the underlying resources of the on-premises standby scheduling cluster and the on-premises primary scheduling cluster are isolated.

[0026] A cloud-based scheduling cluster migration device includes:

[0027] The job scheduling unit is used to initialize the job tasks obtained from the on-premises primary scheduling cluster to the cloud backup scheduling cluster through a scheduling decorator. The cloud backup scheduling cluster and the on-premises primary scheduling cluster have a primary-backup relationship.

[0028] The node execution unit is used to start the cloud-based backup scheduling cluster and determine the execution method corresponding to the node based on the node type of each process in the job task.

[0029] The comparison unit is used to determine whether the job execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster is consistent.

[0030] The identity adjustment unit is used to swap the identities of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster when the job execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster are consistent.

[0031] An electronic device, comprising:

[0032] At least one processing device and a storage device connected to the processing device, wherein:

[0033] The storage device is used to store computer programs;

[0034] The processing device is used to execute the computer program so that the electronic device can implement the cloud-based scheduling cluster scheduling migration method described in any one of the above descriptions.

[0035] A server includes the aforementioned on-premises scheduling cluster scheduling migration device.

[0036] Based on the above technical solution, the solution provided in this embodiment of the invention creates a cloud-based backup scheduling cluster. Then, a scheduling decorator is used to deploy and initialize the same job tasks as the on-premises primary scheduling cluster on the cloud-based backup scheduling cluster, and execution begins. If the cloud-based backup scheduling task fails to execute due to environmental or other factors, it does not affect the execution of the on-premises primary scheduling cluster. Only when both the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster successfully execute the job tasks are their identities swapped, with the cloud-based backup scheduling cluster becoming the primary scheduling cluster and the on-premises primary scheduling cluster becoming the backup scheduling cluster. This ensures that after the task is deployed to the cloud, the cloud-based backup scheduling cluster can successfully execute the scheduling task. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating a cloud-based scheduling cluster migration method disclosed in an embodiment of this application.

[0039] Figure 2 This is a schematic diagram of the structure of the cloud-based scheduling cluster migration device disclosed in the embodiments of this application. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Firstly, we will explain some of the technical terms used in this application:

[0042] Job tasks: A workflow in which task nodes of a series of ETL data flow have dependencies (upstream and downstream) and are scheduled to be executed at fixed times every day or hour.

[0043] Scheduling cluster: A container that carries out scheduled job execution.

[0044] Task migration to the cloud: The scheduling cluster currently running on physical machines needs to be moved to a cloud server for better operation, maintenance and expansion. This process is called task migration to the cloud.

[0045] Cloud-based scheduling clusters (such as the cloud-based backup scheduling cluster mentioned below): For example, a scheduling cluster built on a cloud server.

[0046] On-premises scheduling cluster (e.g., the on-premises main scheduling cluster mentioned below): a scheduling cluster built on physical machines in the company's own data center.

[0047] This invention employs a cloud-based primary scheduling cluster and a cloud-based backup scheduling cluster, along with a scheduling decorator and an underlying data warehouse storage isolation scheme. This allows multiple tasks of the same job to be executed on both the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster. Because the resources of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster are isolated, problems with the cloud-based backup scheduling cluster will not affect the execution of existing services. Furthermore, if a problem occurs when switching to the cloud-based backup scheduling cluster, since the cloud-based cluster is still running and the data has already been produced, only a business environment switch is required, resulting in high fault tolerance. When the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster maintain consistency with the same data warehouse table, a primary / backup switch is performed between the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster.

[0048] See Figure 1 This application provides a method for migrating scheduling in an on-premises scheduling cluster, the method comprising:

[0049] Step S101: Initialize the job tasks obtained from the on-premises main scheduling cluster to the cloud backup scheduling cluster using the scheduling decorator.

[0050] The cloud-based backup scheduling cluster and the on-premises primary scheduling cluster have a primary-backup relationship. When the cloud-based backup scheduling cluster is configured correctly, the execution results of the same job task will be identical when both clusters process the same task. The cloud-based backup scheduling cluster and the on-premises primary scheduling cluster interact through a scheduling decorator. The scheduling decorator, acting as an intermediary layer, can obtain job tasks from the on-premises primary scheduling cluster, initialize the task, and send it to the cloud-based backup scheduling cluster, thus achieving data synchronization between the two clusters. The core of the scheduling decorator can transparently forward scheduling requests (such as task submission and status query) from the on-premises job cluster to the cloud-based backup scheduling cluster, ensuring that the job task configuration and dependency libraries of the cloud-based backup scheduling cluster are fully synchronized with the on-premises primary scheduling cluster during initialization.

[0051] In this step, the job tasks obtained by the cloud-based main scheduling cluster can refer to all cloud-based scheduling job tasks. That is, the job tasks can refer to all job tasks that the cloud-based main scheduling cluster can process, or they can refer to the various job tasks obtained by the cloud-based scheduling cluster during its operation.

[0052] Step S102: Start the cloud-based backup scheduling cluster operation;

[0053] In this step, the backup scheduling cluster is started to execute the job task. When the backup scheduling cluster in the cloud executes the job task obtained in step S101, it will determine whether to rely on the job task node status of the primary scheduling cluster on the cloud based on the lake entry and exit attributes of each node during the execution process. In this scheme, the lake entry node refers to the data access node, which is used to obtain the job task. The lake node refers to the data processing / transformation node of the task data when the scheduling cluster executes the job task. This process is completed within the scheduling cluster. The lake exit node refers to the data output / service node, that is, the node that feeds back the execution result of the job task to the downstream device. This node is only executed by the primary scheduling cluster. The backup scheduling cluster does not need to execute it. However, the backup scheduling cluster will take the execution status of this node in the primary scheduling cluster as the execution status of this node in the backup scheduling cluster. The backup scheduling cluster does not need to actually output the execution result of the job task. This can prevent the data acquisition end from obtaining an execution result separately from the primary scheduling cluster and the backup scheduling cluster. For example, when the execution result of a data output / service node for a job task in the primary on-premises scheduling cluster is "failure," the "failure" value is assigned to the execution result of the same job task's data output / service node in the backup scheduling cluster in the cloud. When the execution result of a data output / service node for a job task in the primary on-premises scheduling cluster is "success," the "success" value is assigned to the execution result of the same job task's data output / service node in the backup scheduling cluster in the cloud, but the backup scheduling cluster in the cloud does not output the execution result of the job task. When the backup scheduling cluster in the cloud is executing a job task, the node type in the job task's process is either an inbound node (data access node) or an inbound node (data processing / conversion node), which can be executed independently without needing to pay attention to the execution results of the inbound nodes and inbound nodes in the primary on-premises scheduling cluster. That is, the processing flow of this node is executed independently in the backup scheduling cluster in the cloud and the primary on-premises scheduling cluster, and the execution status of this node in the backup scheduling cluster in the cloud and the primary on-premises scheduling cluster does not affect each other.

[0054] Step S103: Determine whether the job execution status of the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster is consistent.

[0055] Once all issues causing job execution failures due to environmental or network problems in the cloud-based backup scheduling cluster have been resolved, the execution status of the same job task should be consistent between the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster. This execution status includes the execution status of the nodes entering the lake, the nodes within the lake, and the nodes leaving the lake for each task within the same job task.

[0056] After a task is completed in both the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster, it is determined whether the task was successfully executed. Furthermore, the execution status of the nodes entering the lake, the nodes within the lake, and the nodes leaving the lake in both clusters must remain consistent. If the result is yes, it indicates that the cloud-based backup scheduling cluster can reliably execute this type of task. Otherwise, it indicates that the cloud-based backup scheduling cluster is not yet capable of executing this type of task and further modifications to the cluster are required.

[0057] Step S104: When the job execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster is consistent, the identities of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster are swapped, with the cloud-based backup scheduling cluster becoming the primary scheduling cluster and the cloud-based primary scheduling cluster becoming the backup scheduling cluster.

[0058] When both the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster have successfully executed the job tasks, and the execution status of the nodes entering the lake, the nodes within the lake, and the nodes leaving the lake in the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster are consistent, the cloud-based backup scheduling cluster will be used as the primary cluster, and the on-premises primary scheduling cluster will be used as the backup cluster.

[0059] In this embodiment, the underlying resources of the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster are isolated. With independent databases used for the cloud and on-premises clusters, operations during the cloud migration process (such as data migration, testing, and rollback) only affect the cloud environment. Errors or malfunctions will not lead to incorrect modification or deletion of data in the on-premises database, avoiding the risk of a domino effect. Furthermore, if the cloud cluster crashes due to configuration errors, software vulnerabilities, or attacks, the on-premises cluster can still operate normally, ensuring that core business operations are unaffected.

[0060] As can be seen from the above scheme, after creating the cloud-based backup scheduling cluster, the same job tasks as the on-premises primary scheduling cluster are deployed and initialized on the cloud-based backup scheduling cluster using a scheduling decorator, and then execution begins. At this time, if the cloud-based backup scheduling task fails to execute due to environmental or other factors, it will not affect the execution of the job tasks on the on-premises primary scheduling cluster. Only when both the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster successfully execute the job tasks will their identities be swapped, with the cloud-based backup scheduling cluster becoming the primary scheduling cluster and the on-premises primary scheduling cluster becoming the backup scheduling cluster. During this process, there is sufficient time to correct the cloud-based backup scheduling cluster, and the reliable execution of job tasks and the correct output of business data can be guaranteed.

[0061] In this embodiment, when the cloud-based standby scheduling cluster is executing a certain job task, if the node type of a certain process is a data output / service node, the execution status of the process in the cloud-based main scheduling cluster can be obtained through the scheduling decorator (the execution status includes two states: "success" and "failure"). When the execution status of the process in the cloud-based main scheduling cluster is successful, the execution status of the task in the cloud-based standby scheduling cluster is marked as successful. When the execution status of the process in the cloud-based main scheduling cluster is failed, the execution status of the task in the cloud-based standby scheduling cluster is marked as failed. When the cloud-based main scheduling cluster has not yet completed the process (e.g., not started, currently executing, retrying), the scheduling decorator continues to wait until the process is completed.

[0062] In this embodiment, all job tasks executed by the primary on-premises scheduling cluster can be initialized to both the primary on-premises scheduling cluster and the backup scheduling cluster in the cloud. The primary on-premises scheduling cluster and the backup scheduling cluster execute all job tasks once, and the execution results determine whether the roles of the backup scheduling cluster and the primary on-premises scheduling cluster can be switched. This method requires allowing sufficient testing time. During testing, the primary on-premises scheduling cluster may have difficulty responding to normal scheduling requests, or its response speed to normal scheduling requests may be slow. In this application, the job tasks obtained by the cloud-based backup scheduling cluster can be actual job tasks obtained during the normal operation of the cloud-based primary scheduling cluster. In this case, it is not necessary to reserve separate testing time for the cloud-based primary scheduling cluster and the cloud-based backup scheduling cluster. The testing of the cloud-based backup scheduling cluster can be completed during the normal operation of the cloud-based primary scheduling cluster. At this time, the scheduling decorator initializes the job tasks obtained during the normal operation of the cloud-based primary scheduling cluster to the cloud-based backup scheduling cluster. Then, it is determined whether the job task execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster are consistent. Specifically, when the working time of the cloud-based backup scheduling cluster reaches the first calibration time or the amount of job tasks processed reaches the target amount, if the execution status of each job task of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster are the same, it indicates that the job task execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster are consistent; otherwise, it indicates that the job task execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster are inconsistent. The working time and workload can be set based on experience to ensure that the types of tasks processed by the cloud-based backup scheduling cluster can cover all task types. Although this cloud-based backup scheduling cluster testing method is time-consuming, it does not affect the normal operation of the on-premises primary scheduling cluster.

[0063] In this embodiment, after the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster exchange identities, the cloud-based backup scheduling cluster may still experience abnormal situations under certain circumstances. To address this, in this embodiment, after the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster exchange identities, it is not necessary to immediately shut down the on-premises primary scheduling cluster. Instead, the on-premises primary scheduling cluster is controlled to continue working. At this time, the scheduling decorator synchronizes the job tasks obtained by the cloud-based backup scheduling cluster to the on-premises primary scheduling cluster. The on-premises primary scheduling cluster processes the job tasks but does not output the processing results. Based on the processing results of each job task by the on-premises primary scheduling cluster and the cloud-based backup scheduling cluster, it is determined whether the cloud-based backup scheduling cluster has encountered any abnormalities when executing job tasks within the second calibration period. If an abnormality occurs, it indicates that the cloud-based backup scheduling cluster cannot operate reliably and normally. The identities of the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster are exchanged again, with the on-premises primary scheduling cluster becoming the primary scheduling cluster and the cloud-based backup scheduling cluster becoming the backup scheduling cluster. After the cloud-based backup scheduling cluster is adjusted, step S101 and subsequent steps are executed again. If no abnormalities occur, and the execution results of the job tasks are the same when the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster execute the job tasks, it indicates that the cloud-based backup scheduling cluster can work reliably, and at this time the backup scheduling cluster can be shut down.

[0064] In this embodiment, when there are some errors in the configuration of the cloud-based backup scheduling cluster, it may fail to execute specific job tasks. In this case, the execution result is that the execution status of the job tasks in the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster are inconsistent. In order to facilitate technicians to quickly correct the cloud-based backup scheduling cluster, this embodiment can mark the job tasks in the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster that have inconsistent execution status, and record the execution log of the cloud-based backup scheduling cluster when executing the job tasks. Technicians can quickly determine the cause of the failure and correct the cloud-based scheduling cluster by using the marking results and the execution log of the cloud-based backup scheduling cluster.

[0065] In summary, this application has the following advantages:

[0066] This invention first deploys a backup scheduling cluster in the cloud and initializes the same job tasks as the primary scheduling cluster on the cloud, and starts execution. At this time, even if the scheduling tasks of the backup scheduling cluster in the cloud fail due to environmental factors, it will not affect the operation of the scheduling jobs of the primary scheduling cluster on the cloud. There is sufficient time to handle the failed tasks in the new cloud environment. When the scheduling task status of the backup scheduling cluster in the cloud and the primary scheduling cluster on the cloud are consistent, the primary and backup relationship between the two can be switched, which greatly ensures the stability of scheduling migration.

[0067] Corresponding to the above method, this embodiment also discloses a cloud-based scheduling cluster scheduling migration device. For the specific working content of each unit in the device, please refer to the content of the above method embodiment.

[0068] The following describes the cloud-based scheduling cluster migration device provided in the embodiments of the present invention. The cloud-based scheduling cluster migration device described below and the cloud-based scheduling cluster migration method described above can be referred to and correspond to each other.

[0069] See Figure 2 The device includes:

[0070] The job scheduling unit 10 is used to initialize the job tasks obtained by the on-premises main scheduling cluster to the cloud standby scheduling cluster through the scheduling decorator. The cloud standby scheduling cluster and the on-premises main scheduling cluster have a main-standby relationship.

[0071] The node execution unit 20 is used to start the cloud-based backup scheduling cluster and determine the execution method corresponding to the node based on the node type of each process in the job task.

[0072] The comparison unit 30 is used to determine whether the job execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster is consistent.

[0073] The identity adjustment unit 40 is used to swap the identities of the cloud-based backup scheduling cluster and the cloud-based main scheduling cluster when the job execution status of the cloud-based backup scheduling cluster and the cloud-based main scheduling cluster are consistent.

[0074] An electronic device includes at least one processing unit and a storage unit connected to the processing unit, wherein: the storage unit is used to store a computer program; and the processing unit is used to execute the computer program to enable the electronic device to implement any of the above-described cloud-based scheduling cluster migration methods.

[0075] For example, the processing device is used to: initialize the job tasks obtained by the on-premises primary scheduling cluster to the cloud backup scheduling cluster through a scheduling decorator, wherein the cloud backup scheduling cluster and the on-premises primary scheduling cluster have a primary-backup relationship; start the cloud backup scheduling cluster and determine the execution mode corresponding to the node based on the node type of each process in the job task; determine whether the job task execution status of the cloud backup scheduling cluster and the on-premises primary scheduling cluster is consistent; when the job task execution status of the cloud backup scheduling cluster and the on-premises primary scheduling cluster is consistent, swap the identities of the cloud backup scheduling cluster and the on-premises primary scheduling cluster.

[0076] Corresponding to the above-mentioned device, this application also discloses a server, which includes the above-mentioned on-premises scheduling cluster scheduling migration device. The server can be a cloud server, and the on-premises backup scheduling cluster is located in the cloud server.

[0077] All information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0078] For ease of description, the above system is described by dividing it into various modules based on their functions. Of course, in implementing this invention, the functions of each module can be implemented in one or more software and / or hardware components.

[0079] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0080] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0081] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0082] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0083] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for migrating scheduling clusters in a cloud environment, characterized in that, include: The scheduling decorator initializes the job tasks obtained from the on-premises primary scheduling cluster to the cloud-based backup scheduling cluster, and the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster have a primary-backup relationship. Start the cloud-based backup scheduling cluster and determine the execution method corresponding to the node based on the node type of each process in the job task; Determine whether the execution status of the job tasks in the cloud-based backup scheduling cluster is consistent with that in the on-premises primary scheduling cluster; When the task execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster is consistent, the identities of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster are swapped.

2. The cloud-based scheduling cluster scheduling migration method according to claim 1, characterized in that, Determining the execution method corresponding to a node based on the node type of each process in the task includes: When the node type of the process in the job task is a data access node or a data processing / conversion node, the process will be executed independently in the cloud backup scheduling cluster and the on-premises main scheduling cluster. When the node type of the process in the job task is a data output / service node, the process will be executed independently in the main scheduling cluster on the cloud, and the backup scheduling cluster on the cloud will not execute the process.

3. The cloud-based scheduling cluster migration method according to claim 2, characterized in that, When the node type in the workflow of a job task is a data output / service node, it also includes: The execution status of this process in the on-premises main scheduling cluster is obtained through the scheduling decorator; When the execution status of the process in the primary scheduling cluster under the cloud is successful, the execution status of the task in the backup scheduling cluster on the cloud is marked as successful. When the execution status of the process in the primary scheduling cluster on the cloud is marked as failed, the execution status of the task in the backup scheduling cluster on the cloud is marked as failed.

4. The cloud-based scheduling cluster scheduling migration method according to claim 1, characterized in that, Determining whether the job execution status of the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster is consistent includes: When the working time of the cloud-based backup scheduling cluster reaches the first calibration time or the amount of jobs processed reaches the target amount, the execution status of each job in the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster is the same, indicating that the job execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster is consistent. Otherwise, it indicates that the job execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster is inconsistent.

5. The cloud-based scheduling cluster migration method according to claim 1, characterized in that, After swapping the identities of the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster, the process further includes: Determine whether the cloud-based backup scheduling cluster encounters any abnormalities while executing job tasks within the second calibration period. If an abnormality occurs, swap the identities of the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster again. If no abnormality occurs, and the execution results of the job tasks are the same when the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster execute the job tasks, then the backup scheduling cluster is shut down.

6. The cloud-based scheduling cluster scheduling migration method according to claim 1, characterized in that, After determining whether the job execution status of the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster is consistent, the process includes: The tasks whose execution status is inconsistent between the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster are marked, and the execution log of the cloud-based backup scheduling cluster when executing the tasks is recorded.

7. The cloud-based scheduling cluster scheduling migration method according to claim 1, characterized in that, The underlying resources of the cloud-based backup scheduling cluster and the on-premises primary scheduling cluster are isolated.

8. A cloud-based scheduling cluster migration device, characterized in that, include: The job scheduling unit is used to initialize the job tasks obtained by the on-premises primary scheduling cluster to the cloud backup scheduling cluster through the scheduling decorator. The cloud backup scheduling cluster and the on-premises primary scheduling cluster have a primary-backup relationship. The node execution unit is used to start the cloud-based backup scheduling cluster and determine the execution method corresponding to the node based on the node type of each process in the job task. The comparison unit is used to determine whether the job execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster is consistent. The identity adjustment unit is used to swap the identities of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster when the job execution status of the cloud-based backup scheduling cluster and the cloud-based primary scheduling cluster are consistent.

9. An electronic device, characterized in that, include: At least one processing device and a storage device connected to the processing device, wherein: The storage device is used to store computer programs; The processing device is used to execute the computer program so that the electronic device can implement the cloud-based scheduling cluster scheduling migration method as described in any one of claims 1 to 7.

10. A server, characterized in that, Includes the cloud-based scheduling cluster scheduling migration device as described in claim 8.

Images