Distributed task scheduling methods, apparatus, computer equipment and media

Abstract

This application relates to a distributed task scheduling method, apparatus, computer device, and medium. It pertains to the field of distributed system technology. In this distributed task scheduling method, the original master node device creates task nodes during the execution of a first target task and writes the task information of the first target task into the task nodes. When the original master node fails, a secondary node device becomes the target master node device. The target master node device then scans a preset directory of nodes in the ZooKeeper system. If at least one task node exists under a directory node, task information is read from that at least one task node. The first target task is then restored and executed based on the task information. Because the ZooKeeper system has high availability, it ensures that the data storage of the distributed system is not easily corrupted, thereby guaranteeing the high availability of the distributed system.

Description

Technical Field

[0001] This application relates to the field of distributed systems technology, and in particular to a distributed task scheduling method, apparatus, computer equipment, and medium. Background Technology

[0002] A distributed system comprises multiple computer devices, including master nodes and secondary nodes. The master node receives task requests and creates and executes tasks accordingly. If a master node fails, the distributed system can convert a secondary node into a new master node, which then resumes the tasks that the previous master node had not completed.

[0003] However, the existing data storage systems in distributed systems are prone to single points of failure, which affects the high availability of distributed systems. Summary of the Invention

[0004] Therefore, it is necessary to provide a distributed task scheduling method, apparatus, computer equipment, and medium to address the aforementioned technical problems.

[0005] A distributed task scheduling method, the method comprising:

[0006] After the secondary node device is converted to the target master node device, the preset directory nodes in the Zookeeper system are scanned. If there is at least one task node under the directory node, the task information is read from at least one task node. At least one task node is created by the original master node device during the execution of the first target task, and the task information is the information related to the execution of the first target task written by the original master node device. The first target task is restored according to the task information and executed.

[0007] This application embodiment, through the distributed storage support of the Zookeeper system, enables the target master node device to recover the first target task and continue executing the first target task in the event of a failure of the original master node device in the distributed system, thereby improving the high availability of the distributed system.

[0008] In one embodiment, the method further includes:

[0009] After receiving the task request, a second target task is created and executed according to the task request. During the execution of the second target task, a task node corresponding to the second target task is created under the preset directory node in the Zookeeper system, and the task information corresponding to the second target task is written into the task node corresponding to the second target task.

[0010] By writing the task information of the second target task into the Zookeeper system when the second target task is established, preparations are made for the subsequent recovery and execution of the second target task, thereby improving the high availability of the distributed system.

[0011] In one embodiment, a task node corresponding to the second target task is created under a preset directory node in the Zookeeper system, and the task information corresponding to the second target task is written into the task node corresponding to the second target task. This includes: dividing the second target task into multiple target subtasks, each target subtask being a task at a different stage of the second target task; creating task nodes corresponding to each target subtask under the directory node according to the execution order of each target subtask, and writing the task information corresponding to each target subtask into the task node corresponding to each target subtask.

[0012] By dividing the second target task into multiple independent target subtasks and establishing task nodes for each target subtask, in the event of a failure of the target master node device, the new master node device only needs to restore a portion of the target subtasks of the second target task, rather than restoring and re-executing the entire second target task. This avoids duplicate work in the distributed system and affects the execution efficiency of the task.

[0013] In one embodiment, during the execution of the second target task, a task node corresponding to the second target task is established under a preset directory node in the Zookeeper system, and the task information corresponding to the second target task is written into the task node corresponding to the second target task. This includes: dividing the second target task into multiple target subtasks arranged in the order of execution, each target subtask being a task at a different stage of the second target task; storing the task information corresponding to each target subtask in the target task node; and for each target subtask, when executing the target subtask, establishing a subtask node corresponding to the target subtask in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node, and deleting the subtask node corresponding to the previous target subtask of the target subtask.

[0014] The subtask nodes corresponding to each target subtask are not created all at once. Instead, as the target subtasks are executed sequentially, a subtask node is created for each target subtask that is executed, and the subtask node is deleted after the subtask is completed. This process is repeated sequentially to execute each target subtask. This saves storage space on the ZooKeeper system.

[0015] In one embodiment, the method further includes: during the execution of the second target task, acquiring task status information of the second target task until the second target task is completed; the task status information is used to indicate the execution status of the second target task; and writing the task status information into the task node corresponding to the second target task so as to determine whether the second target task has been completed after the target master node device crashes.

[0016] This application embodiment avoids frequent deletion operations in directory nodes by writing task status information, thereby reducing the data processing volume of the Zookeeper system and improving the working efficiency of the distributed system.

[0017] In one embodiment, writing task status information into the task node corresponding to the second target task includes: establishing a transaction marker under the directory node, the transaction marker including the identifiers of at least two task nodes to which the task status information is to be written; the transaction marker is used to indicate that the write operation is performed synchronously on at least two task nodes corresponding to the identifiers of at least two task nodes; backing up the task information in at least two task nodes to obtain backup data; and writing the task status information into at least two task nodes corresponding to the identifiers of at least two task nodes.

[0018] In one embodiment, the method further includes:

[0019] If the write operation is successful, delete the backup data and transaction markers;

[0020] If the write operation fails, the task information in at least two task nodes corresponding to the identifiers of at least two task nodes is restored based on the backup data, and the transaction marker is deleted.

[0021] By establishing and deleting transaction markers, operations on multiple task nodes are enabled, ensuring the atomicity of operations on the Zookeeper system. This resolves the issue of Zookeeper's inability to support simultaneous transaction operations on multiple task nodes. It also fully supports a high-availability job framework based on the Zookeeper system.

[0022] In one embodiment, the method further includes:

[0023] After the second target task is completed, delete the task node corresponding to the second target task from the directory nodes.

[0024] By deleting the task node corresponding to the completed second objective task, the amount of data storage can be simplified, and the data management burden of the Zookeeper system can be reduced.

[0025] A distributed task scheduling device, the device comprising:

[0026] The scanning module is used to scan the preset directory nodes in the Zookeeper system after the secondary node device is converted to the target master node device;

[0027] The reading module is used to read task information from at least one task node if there is at least one task node under the directory node; wherein, at least one task node is created by the original master node device during the execution of the first target task, and the task information is information related to the execution of the first target task written by the original master node device.

[0028] The recovery module is used to recover the first target task based on the task information and execute the first target task.

[0029] A computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: after converting from a secondary node device to a target master node device, scanning a preset directory node in the Zookeeper system; if at least one task node exists under the directory node, reading task information from the at least one task node; wherein the at least one task node was created by the original master node device during the execution of a first target task, and the task information is information related to the execution of the first target task written by the original master node device; restoring the first target task based on the task information, and executing the first target task.

[0030] A computer-readable storage medium storing a computer program, which, when executed by a processor, performs the following steps: after switching from a secondary node device to a target master node device, scanning a preset directory node in the Zookeeper system; if at least one task node exists under the directory node, reading task information from the at least one task node; wherein the at least one task node was created by the original master node device during the execution of a first target task, and the task information is information related to the execution of the first target task written by the original master node device; restoring the first target task based on the task information, and executing the first target task.

[0031] The aforementioned distributed task scheduling method, apparatus, computer equipment, and medium can ensure the high availability of the distributed system. In this distributed task scheduling method, the original master node device creates task nodes during the execution of the first target task and writes the task information of the first target task into the task nodes. Thus, when the original master node fails, the secondary node device becomes the target master node device. The target master node device then scans a preset directory node in the ZooKeeper system; if at least one task node exists under the directory node, it reads the task information from the at least one task node; it recovers the first target task based on the task information and executes the first target task. Because the ZooKeeper system has high availability, it can ensure that the data storage of the distributed system is not easily invalidated, thereby guaranteeing the high availability of the distributed system. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the implementation environment involved in the distributed task scheduling method in one embodiment;

[0033] Figure 2 This is a schematic diagram of the structure of a master node device;

[0034] Figure 3 This is a schematic diagram of another type of master node device;

[0035] Figure 4 A flowchart illustrating a distributed task scheduling method provided in this application embodiment;

[0036] Figure 5 A flowchart illustrating another distributed task scheduling method provided in this application embodiment;

[0037] Figure 6 A flowchart illustrating another distributed task scheduling method provided in this application embodiment;

[0038] Figure 7 A flowchart illustrating another distributed task scheduling method provided in this application embodiment;

[0039] Figure 8 A flowchart illustrating a method for performing write operations on multiple task nodes in a Zookeeper system, as provided in an embodiment of this application;

[0040] Figure 9 A structural block diagram of a distributed task scheduling device provided in an embodiment of this application;

[0041] Figure 10 This is a structural block diagram of another distributed task scheduling device provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] First, let's explain the architecture of the distributed system involved in this case:

[0044] A distributed system is a cluster of multiple computer devices that communicate with each other via a network. These multiple computer devices collaborate to provide a certain service or resource. Generally, one of the multiple computer devices in a distributed system is the master node, and the remaining computer devices are the secondary nodes.

[0045] Secondly, the working process of the distributed system involved in this case will be explained:

[0046] In a distributed system, the master node device executes tasks and stores task information and execution status information during the execution process.

[0047] When the master node fails, the distributed system can select a new master node from multiple secondary nodes based on pre-defined rules. The new master node can then resume execution of tasks that were not completed by the previous master node, based on stored task information and execution status information.

[0048] In practical applications, new master nodes in distributed systems rely heavily on stored data during recovery tasks, thus the validity of stored data significantly impacts the high availability of the distributed system. However, existing distributed systems are prone to data storage failures, leading to instability in their high availability.

[0049] To address the aforementioned issues, this proposal offers the following technical solution: The master node device writes task information into the ZooKeeper system. Since ZooKeeper is a distributed system, it inherently possesses high availability and is therefore less prone to failure. Thus, if the master node device fails, a new master node device can reliably retrieve the task information needed for recovery from the ZooKeeper system, thereby ensuring the high availability of the distributed system.

[0050] Furthermore, the distributed task scheduling method provided in this application, based on the distributed consistency support of the Zookeeper system, simplifies the consistency processing of high-availability task information across different servers. Moreover, the distributed storage support of the Zookeeper system simplifies the design of synchronizing high-availability task information across different servers. Finally, this application solves the technical problem that the Zookeeper system does not support simultaneous transaction operations on multiple task nodes, fully supporting a high-availability job framework based on the Zookeeper system.

[0051] The following is a brief description of the implementation environment involved in the distributed task scheduling method provided in the embodiments of this application.

[0052] like Figure 1 As shown, the implementation environment includes a server cluster 101 configured with a Zookeeper system and a computer device cluster 102. The computer device cluster 102 is configured with a distributed system, which can apply the distributed scheduling method provided in this application.

[0053] The Zookeeper system provides support for consistency and distributed storage in distributed systems. A server cluster 101 configured with Zookeeper includes a master server and several slave servers. This cluster maintains a file system-like data structure containing multiple directory nodes, each of which can also contain multiple subdirectory nodes. Each directory node or subdirectory node can store data. When the master server in the Zookeeper-configured server cluster 101 fails, a slave server in the cluster can become the new master server to ensure the continued operation of the Zookeeper system.

[0054] The computer equipment cluster 102 includes multiple computer devices, one of which is a master node device 1021, and the remaining computer devices are secondary node devices 1022. The master node device 1021 is connected to the master server in the server cluster 101 configured with the Zookeeper system, so as to perform write and read operations in the Zookeeper system.

[0055] In this embodiment, the master node device can be a server or a terminal. If the master node device is a terminal, its internal structure diagram can be as follows: Figure 2As shown, the master node device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the master node device provides computing and control capabilities. The memory of the master node device includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the master node device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the distributed task scheduling method provided in this application. The master node device may include a liquid crystal display screen or an electronic ink display screen, and the input device of the master node device may be a button, trackball, or touchpad, or it may be an external keyboard, touchpad, or mouse, etc.

[0056] If the master node device is a server, its internal structure diagram can be as follows: Figure 3 As shown, the master node device includes a processor, memory, and a network interface connected via a system bus. The processor of the master node device provides computing and control capabilities. The memory of the master node device includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface of the master node device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a distributed task scheduling method.

[0057] Those skilled in the art will understand that Figure 2 and Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0058] Please refer to Figure 4 It shows a flowchart of a distributed task scheduling method provided in an embodiment of this application, such as... Figure 4 As shown, the distributed task scheduling method may include the following steps:

[0059] Step 401: After switching from a secondary node device to the target primary node device, scan the preset directory nodes in the Zookeeper system.

[0060] First, it should be noted that both the master node and slave node devices configured with a distributed system pre-store the path of the directory node, which is used to point to the preset directory node in the Zookeeper system.

[0061] During the operation of the distributed system, the original master node device can receive task requests and create and execute a first target task based on the task requests. The first target task is an application-type task that the distributed system needs to execute; however, this embodiment does not limit the actual task type of the first target task.

[0062] During the execution of the first objective task, the existing master node device can access the directory node in the Zookeeper system based on the path of the pre-stored directory node, create a task node under that directory node, and then write the task information of the first objective task into the task node. The task information of the first objective task is information related to the execution of the first objective task.

[0063] When the master node device fails, the distributed system can select a new master node device from multiple secondary node devices based on pre-defined rules. This new master node device is referred to as the target master node device in this application. After the selected secondary node device's identity is changed from a secondary node device to the target master node device, the target master node device needs to check whether there are any unfinished tasks from the original master node device. Therefore, the target master node device needs to scan the preset directory nodes in the ZooKeeper system according to the paths of its pre-stored directory nodes.

[0064] Step 402: If there is at least one task node under the directory node, then read the task information from at least one task node.

[0065] Among them, at least one task node is created by the original master node device during the execution of the first target task, and the task information is information related to the execution of the first target task written by the original master node device.

[0066] In one alternative implementation, after scanning the directory nodes, if the target master node device finds at least one task node under the directory node, it indicates that there is a task that the original master node device has not completed. In this case, the target master node device can read the task information from the at least one task node.

[0067] If no task node exists under the directory node, it means that the original master node device has completed the execution of all the first target tasks.

[0068] In one optional implementation, if there is at least one task node under the directory node, it means that the original master node device has previously established the first target task. The target master node device can then read the task information in the at least one task node. Optionally, the task information may include the task status, which indicates whether the first target task has been completed. In this way, the target master node device can determine whether there is a task that the original master node device has not completed based on the task information read from the at least one task node.

[0069] Step 403: Restore the first target task based on the task information and execute the first target task.

[0070] In this embodiment of the application, if the target master node device determines that there is an existing master node device that has not completed a task, it can re-establish the first target task based on the task information and then continue to execute the first target task.

[0071] In this embodiment, the original master node device creates task nodes during the execution of the first target task and writes the task information of the first target task into the task nodes. Thus, when the original master node fails, the secondary node device becomes the target master node device. The target master node device then scans a preset directory node in the ZooKeeper system. If at least one task node exists under the directory node, it reads the task information from the at least one task node. Based on the task information, it restores and executes the first target task. Because the ZooKeeper system has high availability, it can ensure that the data storage of the distributed system is not easily invalidated, thereby guaranteeing the high availability of the distributed system.

[0072] In another embodiment of this application, such as Figure 5 As shown, another distributed scheduling method provided in this application embodiment is illustrated, the method comprising:

[0073] Step 501: After receiving the task request, create a second target task according to the task request and execute the second target task.

[0074] In this embodiment of the application, after receiving a task request, the target master node device can establish a second target task according to the task request and execute the second target task.

[0075] Optionally, in this embodiment of the application, the task request may carry task information, including task attribute information, which is information related to the establishment of the second target task. The target master node device can establish the second target task according to the task information and execute the second target task.

[0076] It should be noted that in the embodiments of this application, the target task established by the original master node device is called the first target task, and the task established by the target master node device is called the second target task. The first target task and the second target task do not have a temporal order relationship.

[0077] Step 502: During the execution of the second target task, a task node corresponding to the second target task is created under the preset directory node in the Zookeeper system, and the task information corresponding to the second target task is written into the task node corresponding to the second target task.

[0078] During the execution of the second target task, the target master node device can access the directory node in the Zookeeper system based on the address of the pre-stored directory node, and then create a task node corresponding to the second target task under that directory node.

[0079] In this context, the task node corresponding to the second target task refers to a task node that contains a task identifier for the second target task. This task identifier can be a task number or a name number. The target master node device can determine the target task corresponding to a task node by reading the task identifier within it.

[0080] After establishing the task node corresponding to the second target task, the target master node device can obtain the task information corresponding to the second target task. The task information corresponding to the second target task includes task attribute information, task name, task identifier, start time, and task status. The task attribute information is related to the establishment of the second target task, and the task status can be used to indicate whether the second target task has been completed.

[0081] In this embodiment, after the target master node device establishes the second target task, it writes the task information of the second target task into the task node corresponding to the second target task under the directory node in the Zookeeper system. This allows the new master node device to obtain the task information of the unfinished second target task from the directory node when the target master node device fails, thereby enabling the recovery of the second target task and ensuring the high availability of the distributed system.

[0082] In one optional implementation, in this embodiment of the application, during the execution of the second target task, after the second target task is completed, the target master node device can delete the task node corresponding to the second target task from the directory node. When a new second target task is available, a new task node corresponding to the new second target task is re-established.

[0083] This implementation ensures that all task nodes under the directory node correspond to the unfinished second target task. This facilitates the new master node device to quickly identify the unfinished second target task when the target master node fails, enabling task recovery and continued execution. This improves the task execution efficiency and high availability of the distributed system.

[0084] In practical applications, since the second target task can be quite large, if one second target task corresponds to one task node, and the target master node device fails before the second target task is completed, the new master node device must completely re-enable the second target task when it resumes operation. This means that any parts already executed by the target master node will be re-executed, leading to duplicate jobs in the distributed system and impacting task execution efficiency. To address this technical problem, in the embodiments of this application, as follows... Figure 6 As shown, a novel distributed task scheduling method is proposed, which includes the following steps:

[0085] Step 601: After receiving the task request, create a second target task according to the task request and execute the second target task.

[0086] Step 602: Divide the second objective task into multiple objective sub-tasks, each of which is a task at a different stage of the second objective task.

[0087] In this embodiment, the user can divide the second target task into multiple independently designed target sub-tasks through the target master node device. Each target sub-task is a task of a different stage of the second target task. Each stage is the smallest granularity supported by high-availability jobs.

[0088] Step 603: Under the directory node, create task nodes corresponding to each target subtask according to the execution order of each target subtask, and write the task information corresponding to each target subtask into the task node corresponding to each target subtask.

[0089] The target master node device can create multiple task nodes under the directory node. Each task node is used to store the task information of a target subtask. Each target subtask writes its task information into its corresponding task node in the order of execution.

[0090] Optionally, the task information corresponding to the target subtask may include information such as the second target task identifier, the target subtask identifier, the execution time, and the execution status.

[0091] Optionally, in this embodiment of the application, after each target subtask is executed, the target master node device can delete the task node corresponding to the target subtask from the directory node, so that all task nodes existing in the directory node are task nodes corresponding to the target subtasks that have not yet been executed.

[0092] Optionally, in this embodiment, after each target master node device completes the execution of a target subtask, it can write the task status information of the target subtask to the corresponding task node. This task status information can be used to indicate whether the target subtask has finished executing. Thus, the task nodes existing under the directory node include both the task nodes corresponding to the target subtasks that have finished executing and the task nodes corresponding to the target subtasks that have not yet finished executing.

[0093] Step 604: In the event of a target master node failure, the new master node resumes the unfinished target subtasks from the directory nodes in the Zookeeper system.

[0094] If the target master node fails, the distributed system can select a new master node from multiple secondary nodes. This new master node scans the directory nodes, then resumes any unfinished target subtasks and continues execution. In this way, the new master node only needs to resume a portion of the target subtasks of the second target task, rather than resuming and re-executing the entire second target task, thus avoiding redundant work in the distributed system and improving task execution efficiency.

[0095] In one alternative implementation, since creating task nodes corresponding to all target subtasks in the Zookeeper system when executing the second target task would consume a significant amount of storage space in the Zookeeper system, this application embodiment provides another implementation to save storage space in the Zookeeper system, including:

[0096] The second objective task is divided into multiple target subtasks arranged in the order of execution. Each target subtask is a task at a different stage of the second objective task. The task information corresponding to each target subtask is stored in the target task node. For each target subtask, when executing the target subtask, a subtask node corresponding to the target subtask is created in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node, and the subtask node corresponding to the previous target subtask is deleted.

[0097] In this embodiment of the application, after the second target task is divided into multiple target subtasks, a task node corresponding to each target subtask is not created in the Zookeeper system. Instead, all task information of all target subtasks is stored in the target task node, and can be stored in the order of execution.

[0098] Then, each target subtask is executed sequentially according to the execution order. When the k-th target subtask is executed, the task information corresponding to the k-th target subtask can be obtained from the target task node, and the subtask node corresponding to the k-th target subtask can be established in the Zookeeper system. Then, the execution of the k-th target subtask begins.

[0099] In this way, the subtask nodes corresponding to each target subtask are not created all at once. Instead, as the target subtasks are executed sequentially, a subtask node is created for each target subtask that is executed, and the subtask node is deleted after the subtask is completed. This process is repeated to execute each target subtask sequentially, thereby saving storage space on the ZooKeeper system.

[0100] In this embodiment of the application, after the subtask node corresponding to the k-th target subtask is established, the subtask node corresponding to the previous target subtask, i.e., the (k-1)-th target subtask, is deleted. The (k-1)-th target subtask is a target subtask that has already been executed, which clearly distinguishes target subtasks that have been executed and avoids confusion.

[0101] In the embodiments of this application, such as Figure 7 As shown in the figure, this application provides a new distributed task scheduling method, which includes the following:

[0102] Step 701: During the execution of the second target task, obtain the task status information of the second target task until the second target task is completed.

[0103] In this embodiment of the application, during the execution of the second target task, the target master node device can periodically acquire the task status information of the second target task. The task status information indicates the execution status of the second target task. Optionally, the task status information may include a task name, task identifier, task stage identifier, execution status, task stage start time, or task stage end time, wherein the execution status can be either "in execution" or "execution completed."

[0104] Optionally, if the task status information includes the task phase end time, it indicates that the second target task has been completed; if the task status information does not include the task phase end time, it indicates that the second target task has not been completed.

[0105] Step 702: Write the task status information into the task node corresponding to the second target task so as to determine whether the second target task has been completed after the target master node device crashes.

[0106] In this embodiment, the target master node device can write task status information into the task node corresponding to the second target task. If the target master node device crashes, the new master node device can scan the directory nodes to determine whether the second target task has finished execution based on the task status information of the second target task in the task node corresponding to the second target task.

[0107] In this embodiment of the application, by writing the task status information corresponding to the second target task into the task node corresponding to the second target task, it is more clearly determined whether the second target task has been completed, so that when the target master node device crashes, the new master node device can quickly restore the second target task that has not been completed.

[0108] In practical applications, there may be situations where the data volume of task information corresponding to the second target task is relatively large, or the data volume of task information corresponding to certain target subtasks is relatively large. Since the data storage capacity of each task node in the ZooKeeper system is limited, if the data volume of task information exceeds the data storage capacity of the task node, the task information needs to be split.

[0109] In this embodiment of the application, when the target master node device establishes a task node corresponding to the second target task, it can detect whether the data volume of the task information corresponding to the second target task is greater than the data volume threshold. If it is greater, multiple task nodes corresponding to the data volume of the task information of the second target task are established. These multiple task nodes can be used to store the task information corresponding to the same second target task.

[0110] Correspondingly, the target master node device can also detect whether the data volume of the task information of each target subtask is greater than the data volume threshold. If the data volume of the task information of a certain target subtask is greater than the data volume threshold, multiple task nodes corresponding to the data volume of the task information of that target subtask are established. These multiple task nodes can be used to store the task information corresponding to the same target subtask.

[0111] However, in practical applications, writing task information corresponding to a second task or a target subtask to different task nodes can introduce new transactional issues.

[0112] The following example illustrates the scenario of writing task information corresponding to a second task to different task nodes. Existing ZooKeeper systems do not support simultaneous transaction operations on multiple task nodes. Therefore, the target master node cannot synchronously perform write or read operations on multiple task nodes. This leads to a situation where, when the target master node performs a write operation on the task information of the same second target task, it only performs the write operation on a portion of the multiple task nodes corresponding to the second target task, while not performing the write operation on the other portion. Furthermore, the existing ZooKeeper system cannot identify which task nodes performed write operations and which did not, thus causing difficulties for data management within the ZooKeeper system.

[0113] To address the aforementioned technical problems, embodiments of this application provide an implementation method that allows writing to multiple task nodes in a Zookeeper system, such as... Figure 8 As shown, Figure 8 A flowchart illustrating a method for performing write operations on multiple task nodes in a Zookeeper system, as provided in this application embodiment, is included. The method comprises the following steps:

[0114] Step 801: Create a transaction marker under the directory node.

[0115] In this embodiment of the application, when there are multiple task nodes corresponding to the second target task corresponding to the task status information, the target master node device needs to establish a transaction marker first when writing the target status information. The transaction marker includes the identifiers of at least two task nodes to which the task status information is to be written. The transaction marker is used to indicate that the write operation is performed synchronously on at least two task nodes corresponding to the identifiers of at least two task nodes.

[0116] Step 802: Back up the task information in at least two task nodes to obtain backup data.

[0117] After establishing the transaction markers, the target master node device must explicitly perform write operations on at least two task nodes corresponding to the identifiers of at least two task nodes to avoid missing some task nodes.

[0118] It should be noted that the data in the task nodes under the directory nodes in the Zookeeper system does not support update operations, only overwrite operations. To avoid data storage anomalies caused by write operations, in this embodiment, when performing a write operation, it is necessary to first save the original data in at least two task nodes; the backup data is the original data.

[0119] Step 803: Write the task status information into at least two task nodes corresponding to the identifiers of at least two task nodes.

[0120] The target master node device can identify at least two task nodes under the directory node based on the identifiers of at least two task nodes. Then, write operations are performed on each of the at least two task nodes.

[0121] Step 804: If the write is successful, delete the backup data and transaction marker.

[0122] In this embodiment, a successful write operation is indicated when the write operation has been completed on at least two task nodes. If the write is successful, the backup data is no longer needed and is therefore deleted. Furthermore, a successful write operation signifies that the transaction of writing task status information to the task node has been completed, so the transaction marker is also deleted.

[0123] Step 805: If the write operation fails, restore the task information in at least two task nodes corresponding to the identifiers of at least two task nodes based on the backup data, and delete the transaction marker.

[0124] If the write operation fails, it means that at least two task nodes have not completed the transaction of writing task status information. To ensure the accuracy of the data in these at least two task nodes, the backup data is re-stored into these at least two task nodes to guarantee the accuracy of the task information. At the same time, deleting the transaction marker indicates that this write operation has ended.

[0125] In this embodiment, by establishing and deleting transaction markers, simultaneous transaction operations on multiple task nodes are achieved, ensuring the atomicity of operations on the Zookeeper system. This solves the problem that the Zookeeper system does not support simultaneous transaction operations on multiple task nodes. It fully supports a high-availability job framework based on the Zookeeper system.

[0126] It should be understood that, although Figures 4 to 8 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 4 to 8 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0127] In one embodiment, such as Figure 9As shown, a distributed task scheduling device 900 is provided, including: a scanning module 901, a reading module 902, and a recovery module 903, wherein:

[0128] The scanning module 901 is used to scan the preset directory nodes in the Zookeeper system after the device is converted from a secondary node device to a target master node device.

[0129] The reading module 902 is used to read task information from at least one task node if there is at least one task node under the directory node; wherein, at least one task node is created by the original master node device during the execution of the first target task, and the task information is information related to the execution of the first target task written by the original master node device.

[0130] Recovery module 903 is used to recover the first target task based on the task information and execute the first target task.

[0131] In one embodiment, such as Figure 10 As shown, a distributed task scheduling device 1000 is provided, which further includes:

[0132] The receiving module 1001 is used to, after receiving a task request, establish a second target task according to the task request and execute the second target task;

[0133] Module 1002 is used to create a task node corresponding to the second target task under a preset directory node in the Zookeeper system during the execution of the second target task, and to write the task information corresponding to the second target task into the task node corresponding to the second target task.

[0134] In one embodiment, the establishment module 1002 is specifically used for:

[0135] The second objective task is divided into multiple objective sub-tasks, and each objective sub-task is a task at a different stage of the second objective task.

[0136] Under the directory node, create task nodes corresponding to each target subtask according to the execution order of each target subtask, and write the task information corresponding to each target subtask into the task node corresponding to each target subtask.

[0137] In one embodiment, the establishment module 1002 is specifically used for:

[0138] The second objective task is divided into multiple sub-tasks arranged in the order of execution, and each sub-task is a task at a different stage of the second objective task.

[0139] Save the task information corresponding to each target subtask in the target task node;

[0140] For each target subtask, when executing the target subtask, a subtask node corresponding to the target subtask is created in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node, and the subtask node corresponding to the previous target subtask is deleted.

[0141] In one embodiment, the establishment module 1002 is specifically used for:

[0142] During the execution of the second objective task, the task status information of the second objective task is acquired until the execution of the second objective task is completed; the task status information is used to indicate the execution status of the second objective task.

[0143] The task status information is written into the task node corresponding to the second target task so that it can be determined whether the second target task has been completed after the target master node device fails.

[0144] In one embodiment, the establishment module 1002 is specifically used for:

[0145] A transaction marker is created under the directory node. The transaction marker includes the identifiers of at least two task nodes to which task status information is to be written. The transaction marker is used to indicate that the write operation is performed synchronously on at least two task nodes corresponding to the identifiers of at least two task nodes.

[0146] Back up the task information in at least two task nodes to obtain backup data;

[0147] Write the task status information into at least two task nodes corresponding to the identifiers of at least two task nodes.

[0148] In one embodiment, the establishment module 1002 is specifically used for:

[0149] If the write operation is successful, delete the backup data and transaction markers;

[0150] If the write operation fails, the task information in at least two task nodes corresponding to the identifiers of at least two task nodes is restored based on the backup data, and the transaction marker is deleted.

[0151] In one embodiment, the establishment module 1002 is specifically used for:

[0152] After the second target task is completed, delete the task node corresponding to the second target task from the directory nodes.

[0153] Specific limitations regarding the distributed task scheduling device can be found in the limitations of the distributed task scheduling method above, and will not be repeated here. Each module in the aforementioned distributed task scheduling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0154] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0155] After transitioning from a secondary node device to the target primary node device, scan the preset directory nodes in the Zookeeper system;

[0156] If there is at least one task node under the directory node, then read the task information from at least one task node; wherein, at least one task node is created by the original master node device during the execution of the first target task, and the task information is information related to the execution of the first target task written by the original master node device.

[0157] Restore the first target task based on the task information and execute the first target task.

[0158] In one embodiment, the processor may also perform the following steps when executing a computer program:

[0159] After receiving the task request, a second target task is created based on the task request, and the second target task is executed.

[0160] During the execution of the second objective task, a task node corresponding to the second objective task is created under the preset directory node in the Zookeeper system, and the task information corresponding to the second objective task is written into the task node corresponding to the second objective task.

[0161] In one embodiment, the processor may also perform the following steps when executing a computer program:

[0162] The second objective task is divided into multiple objective sub-tasks, and each objective sub-task is a task at a different stage of the second objective task.

[0163] Under the directory node, create task nodes corresponding to each target subtask according to the execution order of each target subtask, and write the task information corresponding to each target subtask into the task node corresponding to each target subtask.

[0164] In one embodiment, the processor may also perform the following steps when executing a computer program:

[0165] The second objective task is divided into multiple sub-tasks arranged in the order of execution, and each sub-task is a task at a different stage of the second objective task.

[0166] Save the task information corresponding to each target subtask in the target task node;

[0167] For each target subtask, when executing the target subtask, a subtask node corresponding to the target subtask is created in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node, and the subtask node corresponding to the previous target subtask is deleted.

[0168] In one embodiment, the processor may also perform the following steps when executing a computer program:

[0169] During the execution of the second objective task, the task status information of the second objective task is acquired until the execution of the second objective task is completed; the task status information is used to indicate the execution status of the second objective task.

[0170] The task status information is written into the task node corresponding to the second target task so that it can be determined whether the second target task has been completed after the target master node device fails.

[0171] In one embodiment, the processor may also perform the following steps when executing a computer program:

[0172] A transaction marker is created under the directory node. The transaction marker includes the identifiers of at least two task nodes to which task status information is to be written. The transaction marker is used to indicate that the write operation is performed synchronously on at least two task nodes corresponding to the identifiers of at least two task nodes.

[0173] Back up the task information in at least two task nodes to obtain backup data;

[0174] Write the task status information into at least two task nodes corresponding to the identifiers of at least two task nodes.

[0175] In one embodiment, the processor may also perform the following steps when executing a computer program:

[0176] If the write operation is successful, delete the backup data and transaction markers;

[0177] If the write operation fails, the task information in at least two task nodes corresponding to the identifiers of at least two task nodes is restored based on the backup data, and the transaction marker is deleted.

[0178] In one embodiment, the processor may also perform the following steps when executing a computer program:

[0179] After the second target task is completed, delete the task node corresponding to the second target task from the directory nodes.

[0180] The computer device provided in this application embodiment has a similar implementation principle and technical effect to the above method embodiment, and will not be described again here.

[0181] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0182] After transitioning from a secondary node device to the target primary node device, scan the preset directory nodes in the Zookeeper system;

[0183] If there is at least one task node under the directory node, then read the task information from at least one task node; wherein, at least one task node is created by the original master node device during the execution of the first target task, and the task information is information related to the execution of the first target task written by the original master node device.

[0184] Restore the first target task based on the task information and execute the first target task.

[0185] In one embodiment, the computer program, when executed by a processor, may also perform the following steps:

[0186] After receiving the task request, a second target task is created based on the task request, and the second target task is executed.

[0187] During the execution of the second objective task, a task node corresponding to the second objective task is created under the preset directory node in the Zookeeper system, and the task information corresponding to the second objective task is written into the task node corresponding to the second objective task.

[0188] In one embodiment, the computer program, when executed by a processor, may also perform the following steps:

[0189] The second objective task is divided into multiple objective sub-tasks, and each objective sub-task is a task at a different stage of the second objective task.

[0190] Under the directory node, create task nodes corresponding to each target subtask according to the execution order of each target subtask, and write the task information corresponding to each target subtask into the task node corresponding to each target subtask.

[0191] In one embodiment, the computer program, when executed by a processor, may also perform the following steps:

[0192] The second objective task is divided into multiple sub-tasks arranged in the order of execution, and each sub-task is a task at a different stage of the second objective task.

[0193] Save the task information corresponding to each target subtask in the target task node;

[0194] For each target subtask, when executing the target subtask, a subtask node corresponding to the target subtask is created in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node, and the subtask node corresponding to the previous target subtask is deleted.

[0195] In one embodiment, the computer program, when executed by a processor, may also perform the following steps:

[0196] During the execution of the second objective task, the task status information of the second objective task is acquired until the execution of the second objective task is completed; the task status information is used to indicate the execution status of the second objective task.

[0197] The task status information is written into the task node corresponding to the second target task so that it can be determined whether the second target task has been completed after the target master node device fails.

[0198] In one embodiment, the computer program, when executed by a processor, may also perform the following steps:

[0199] A transaction marker is created under the directory node. The transaction marker includes the identifiers of at least two task nodes to which task status information is to be written. The transaction marker is used to indicate that the write operation is performed synchronously on at least two task nodes corresponding to the identifiers of at least two task nodes.

[0200] Back up the task information in at least two task nodes to obtain backup data;

[0201] Write the task status information into at least two task nodes corresponding to the identifiers of at least two task nodes.

[0202] In one embodiment, the computer program, when executed by a processor, may also perform the following steps:

[0203] If the write operation is successful, delete the backup data and transaction markers;

[0204] If the write operation fails, the task information in at least two task nodes corresponding to the identifiers of at least two task nodes is restored based on the backup data, and the transaction marker is deleted.

[0205] In one embodiment, the computer program, when executed by a processor, may also perform the following steps:

[0206] After the second target task is completed, delete the task node corresponding to the second target task from the directory nodes.

[0207] The computer-readable storage medium provided in this embodiment is similar in principle and technical effect to the method embodiment described above, and will not be repeated here.

[0208] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0209] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0210] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A distributed task scheduling method, characterized in that, The method includes: After transitioning from a secondary node device to the target primary node device, scan the preset directory nodes in the Zookeeper system; If there is at least one task node under the directory node, then task information is read from the at least one task node; wherein, the at least one task node is created by the original master node device during the execution of the first target task, and the task information is information related to the execution of the first target task written by the original master node device. The first target task is restored and executed based on the task information. Upon receiving a task request, a second target task is created based on the task request, and the second target task is executed. During the execution of the second target task, the second target task is divided into multiple target subtasks. Under the directory node, task nodes corresponding to each target subtask are created according to their execution order, and the task information corresponding to each target subtask is written into the corresponding task node; or... The second target task is divided into multiple target subtasks arranged in the order of execution, and the task information corresponding to each target subtask is stored in the target task node. For each target subtask, when executing the target subtask, a subtask node corresponding to the target subtask is created in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node, and the subtask node corresponding to the previous target subtask is deleted. Each target subtask is a task at a different stage of the second target task.

2. The method according to claim 1, characterized in that, The method further includes: During the execution of the second target task, the task status information of the second target task is acquired until the second target task is completed; the task status information is used to indicate the execution status of the second target task. The task status information is written into the task node corresponding to the second target task, so as to determine whether the second target task has been completed after the target master node device crashes.

3. The method according to claim 2, characterized in that, The step of writing the task status information into the task node corresponding to the second target task includes: A transaction marker is established under the directory node. The transaction marker includes the identifiers of at least two task nodes to which the task status information is to be written. The transaction marker is used to indicate that the write operation is performed synchronously on the at least two task nodes corresponding to the identifiers of the at least two task nodes. Back up the task information in the at least two task nodes to obtain backup data; The task status information is written into the at least two task nodes corresponding to the identifiers of the at least two task nodes.

4. The method according to claim 3, characterized in that, The method further includes: If the write operation is successful, then delete the backup data and the transaction flag; If the write operation fails, the task information in the at least two task nodes corresponding to the identifiers of the at least two task nodes is restored according to the backup data, and the transaction marker is deleted.

5. The method according to claim 1, characterized in that, The method further includes: After the second target task is completed, the task node corresponding to each target subtask is deleted from the directory node.

6. The method according to claim 1, characterized in that, The task information corresponding to the target subtask includes the identifier of the second target task, the identifier of the target subtask, the execution time, and the execution status.

7. The method according to claim 1, characterized in that, The method further includes: If the task node does not exist under the directory node, then it is determined that all the first target tasks corresponding to the original master node device have been completed.

8. A distributed task scheduling device, characterized in that, The device includes: The scanning module is used to scan the preset directory nodes in the Zookeeper system after the secondary node device is converted to the target master node device; The reading module is used to read task information from at least one task node if there is at least one task node under the directory node; wherein, the at least one task node is created by the original master node device during the execution of the first target task, and the task information is information related to the execution of the first target task written by the original master node device. The recovery module is used to recover the first target task based on the task information and execute the first target task; The receiving module is used to establish a second target task based on the task request after receiving the task request, and to execute the second target task. A module is configured to, during the execution of the second target task, divide the second target task into multiple target subtasks, create task nodes corresponding to each target subtask under the directory node according to the execution order of the target subtasks, and write the task information corresponding to each target subtask into the task node corresponding to each target subtask; or... The second target task is divided into multiple target subtasks arranged in the order of execution, and the task information corresponding to each target subtask is stored in the target task node. For each target subtask, when executing the target subtask, a subtask node corresponding to the target subtask is created in the Zookeeper system based on the task information corresponding to the target subtask stored in the target task node, and the subtask node corresponding to the previous target subtask is deleted. Each target subtask is a task at a different stage of the second target task.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images