Batch job scheduling method, device and job processing system

Abstract

The disclosure provides a batch job scheduling method and device and a job processing system, which can be applied to the technical field of big data. The method is applied to a job processing system, the job processing system comprising a coordination service module and a business system for processing batch jobs, each business system utilizing at least one service instance to execute a plurality of tasks in a batch job. The method comprises: obtaining task information related to each business system; the task information comprising a batch job identifier of a batch job to which the task belongs, a task identifier of the task, a dependency relationship between the tasks, and a service instance identifier of the service instance; creating a directory tree according to the task information, the directory tree comprising a batch job node corresponding to the batch job identifier, a task node corresponding to the task identifier, and a service instance node corresponding to the service instance identifier, and the value of the task node being related to the dependency relationship; and scheduling the task nodes based on the directory tree to execute the tasks in the batch job.

Description

Technical Field

[0001] This disclosure relates to the field of big data, specifically to the field of data preprocessing technology, and more specifically to a batch job scheduling method, job processing system, batch job scheduling device, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] In enterprise production, batch operations are frequently required, such as end-of-day batch processing and year-end closing in banking. Batch operations consist of multiple tasks, and these tasks have a sequential execution relationship. For example, a batch operation might include task A, task B, and task C, where task A depends on tasks B and C. Task A can only be executed after tasks B and C are completed. These tasks can form a Directed Acyclic Graph (DAG). For instance, the DAG formed by the aforementioned batch tasks includes nodes A, B, and C, a first edge, and a second edge. Node A corresponds to task A, node B corresponds to task B, node C corresponds to task C, the first edge points from node B to node A, and the second edge points from node C to node A. Summary of the Invention

[0003] In view of the above problems, this disclosure provides a batch job scheduling method, a job processing system, a batch job scheduling device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] According to a first aspect of this disclosure, a batch job scheduling method is provided, applied to a job processing system, the job processing system including a coordination service module and at least one business system for processing batch jobs, each business system utilizing at least one service instance to execute multiple tasks in the batch job; the method includes: obtaining task information related to each business system; the task information including a batch job identifier of the batch job to which the task belongs, a task identifier of the task, dependencies between tasks, and a service instance identifier of the service instance; creating a directory tree based on the task information, the directory tree including a batch job node corresponding to the batch job identifier, a task node corresponding to the task identifier, and a service instance node corresponding to the service instance identifier, wherein the value of the task node is related to the dependencies; and scheduling the task nodes based on the directory tree to execute tasks in the batch job.

[0005] According to another embodiment of this disclosure, creating a directory tree based on the task information includes: creating a service instance node based on the service instance identifier of a service instance related to the business system; creating a parent node of the service instance node based on the task identifier of the task executed by the business system to obtain a task node; determining the task identifier of other tasks that the task depends on as the value of the task node; and creating a parent node of the task node based on the batch job identifier of the batch job to which the task belongs to obtain a batch job node.

[0006] According to another embodiment of this disclosure, the method further includes: after creating the directory tree, obtaining the status information of the service instance corresponding to the service instance node in the directory tree; and if it is determined that the service instance is in a shutdown state, deleting the node related to the service instance from the directory tree.

[0007] According to another embodiment of this disclosure, when the service instance and the task are respectively related to the same business system, the service instance node corresponding to the service instance is a child node of the task node corresponding to the task; the step of deleting the node related to the service instance from the directory tree includes: deleting the service instance node corresponding to the service instance from the directory tree; and deleting the task node when it is determined that the number of service instance nodes that satisfy the parent-child relationship with the task node is 0.

[0008] According to another embodiment of this disclosure, the method further includes: generating directed acyclic graph information based on the directory tree; wherein the directed acyclic graph information includes nodes and edges, the nodes in the directed acyclic graph information correspond one-to-one with the task nodes in the directory tree, and the edges in the directed acyclic graph information indicate the dependencies between nodes.

[0009] According to another embodiment of this disclosure, the batch job includes multiple task groups, each task group including a first task and a second task dependent on the first task; the scheduling of the task nodes based on the directory tree includes: for each task group, performing the following operations: obtaining the execution status of the first task; updating the value of a first status node in the directory tree according to the execution status, the first status node being a child node of the first task node corresponding to the first task; and triggering a listener registered by the second task when the value of the first status node changes, so that the second task uses the listener to determine the value of the first status node, and starts executing the second task when all tasks dependent on the second task have been executed successfully.

[0010] Another aspect of this disclosure provides a job processing system, including at least one business system and a coordination service module. Each business system utilizes at least one service instance to execute multiple tasks in a batch job. Each business system is configured to scan its own multiple tasks upon startup, obtain task information, and send the task information. The coordination service module is configured to execute the batch job scheduling method described above.

[0011] According to another embodiment of this disclosure, the batch job includes multiple task groups, each task group including a first task and a second task dependent on the first task; the coordination service module is further configured to: obtain the execution status of the first task; update the value of a first status node in a directory tree according to the execution status, the first status node being a child node of a first task node corresponding to the first task; trigger a listener registered by the second task when the value of the first status node changes; the business system is further configured to: register the listener for listening to the first status node with the coordination service module through the first task; in response to the listener being triggered, determine the value of the status node using the listener, and start executing the second task when all tasks dependent on the second task have been executed successfully.

[0012] Another aspect of this disclosure provides a batch job scheduling apparatus applied to a job processing system. The job processing system includes a coordination service module and at least one business system for processing batch jobs. Each business system utilizes at least one service instance to execute multiple tasks within the batch job. The apparatus includes an acquisition module, a creation module, and a scheduling module. The acquisition module acquires task information from each business system. The task information includes a batch job identifier of the batch job to which the task belongs, a task identifier of the task, dependencies between tasks, and service instance identifiers of the service instances. The creation module creates a directory tree based on the task information. The directory tree includes batch job nodes, task nodes, and service instance nodes corresponding to the batch job identifier, the task identifier, and the service instance identifier, respectively, and the value of the task node is related to the dependencies. The scheduling module schedules the task nodes based on the directory tree to execute tasks within the batch job.

[0013] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the batch job scheduling method described above.

[0014] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the batch job scheduling method described above.

[0015] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the batch job scheduling method described above.

[0016] According to the batch job scheduling method, job processing system, batch job scheduling device, electronic device, computer-readable storage medium and computer program product provided in this disclosure, since task information related to each business system is obtained, a directory tree is created based on the task information, and tasks are scheduled based on the directory tree, it is not necessary to manually organize and maintain the DAG corresponding to the batch job using a batch job maintenance center. This achieves the technical effect of automating and simplifying the organization and maintenance of the batch job DAG, and avoiding errors caused by human operation. Attached Figure Description

[0017] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 This diagram illustrates an application scenario of the batch job scheduling method according to embodiments of the present disclosure.

[0019] Figure 2 A flowchart illustrating a batch job scheduling method according to an embodiment of the present disclosure is shown schematically.

[0020] Figure 3 A schematic diagram of a directory tree according to an embodiment of the present disclosure is shown;

[0021] Figure 4 A schematic flowchart illustrating the execution status of a registration update task according to an embodiment of the present disclosure is shown.

[0022] Figure 5 A schematic diagram illustrating responsive scheduling according to an embodiment of the present disclosure is shown.

[0023] Figure 6 A schematic block diagram of a batch job scheduling apparatus according to an embodiment of the present disclosure is shown; and

[0024] Figure 7 A block diagram of an electronic device suitable for implementing a batch job scheduling method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0025] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0029] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0030] In the technical solution disclosed herein, the acquisition, collection, storage, use, processing, transmission, provision, disclosure, and application of data all comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0031] To facilitate understanding of the technical solutions provided in this disclosure, some technical terms are explained below.

[0032] A Directed Acyclic Graph (DAG) is a type of graph in graph theory that has a direction and no cycles. A DAG consists of vertices and edges.

[0033] Batch jobs consist of multiple tasks. By orchestrating the dependencies between these tasks, they are executed in a predetermined order to complete a specific function. These dependencies satisfy a Directed Acyclic Graph (DAG), and each task within a batch job corresponds to a node in the DAG. Furthermore, in practical applications, a batch job can be executed jointly by multiple business systems.

[0034] Task: A relatively independent piece of program in a batch job, used to complete a specific function. Multiple tasks that depend on each other can form a batch job.

[0035] Decentralization: In an organization or network, individual nodes (for example, nodes can represent the tasks described above) are highly autonomous and equal, without the need for a centralized node to coordinate and manage them.

[0036] Dependency scheduling: scheduling is based on the node dependencies in the DAG, meaning that execution will only be triggered after all dependent tasks have been successfully executed.

[0037] XXL-Job: An open-source distributed scheduling platform consisting of a scheduling center and executors. Tasks are configured at the scheduling center, which then schedules the executors to execute the tasks.

[0038] Reactive programming refers to a programming method that uses event listeners to process events in real time as they change.

[0039] In related technologies, staff can manually organize and maintain DAGs using a batch job maintenance center. For example, a batch job maintenance center can be the XXL-Job mentioned above. This center has a console where staff manually configure tasks and their dependencies to obtain the DAG. Furthermore, when tasks change—such as adding or removing tasks, or modifying dependencies—the DAG needs to be maintained synchronously.

[0040] However, in large-scale enterprise-level production, a DAG often has dozens or even hundreds of nodes, and the tasks corresponding to these nodes can be distributed across multiple business systems. For example, a batch job corresponding to a DAG may consist of 300 tasks, which are executed by three business systems: the first business system executes 100 tasks, the second business system executes 70 tasks, and the third business system executes 130 tasks.

[0041] Therefore, when using the aforementioned technologies, manually organizing and maintaining a DAG is a tedious and mechanical task when there are many tasks, and it is prone to errors.

[0042] This disclosure provides a batch job scheduling method that enables tasks to have self-organizing capabilities. When a task node starts, it completes self-registration, thereby forming information describing a DAG. This automates and simplifies the organization and maintenance of the batch job DAG, avoiding errors caused by human operation.

[0043] Figure 1 The diagram illustrates an application scenario of the batch job scheduling method according to an embodiment of the present disclosure.

[0044] like Figure 1 As shown, the application scenarios according to this embodiment may include networks, business systems 101 and 102, and device 103.

[0045] A network is a medium used to provide a communication link between business systems 101, 102 and device 103. A network can include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0046] Business system 101 can be used to execute multiple tasks, such as tasks A1, A2, ..., An. Business system 102 can be used to execute multiple tasks, such as tasks B1, B2, ..., Bm. Business systems 101 and 102 can interact with device 103 via a network to receive or send messages, etc. Business systems 101 and 102 are used to process batch tasks, and each business system can utilize at least one service instance to execute multiple tasks in the batch.

[0047] Business systems 101 and 102 can register with device 103 upon startup. For example, business systems 101 and 102 scan their multiple tasks upon startup to obtain task information, and then send their respective registration information to device 103. The registration information may include task information. Business systems 101 and 102 can also deregister with device 103 when they are shut down.

[0048] Device 103 can integrate a coordination service module, such as Zookeeper. Zookeeper is a distributed, open-source distributed application coordination service that provides functions such as configuration management, name service, distributed locks, and cluster management.

[0049] Device 103 can be a server or server cluster providing various services, such as a backend management server for maintaining task information related to business systems 101 and 102 (this is just an example). The backend management server can analyze and process received registration information and other data, and organize and store the task information in the registration information as a directory tree. Device 103 can also delete information related to business systems 101 and 102 from the directory tree after they are shut down and deregistered. Furthermore, device 103 can dynamically generate a Directed Acyclic Graph (DAG) based on the directory tree and schedule tasks based on the directory tree.

[0050] It should be noted that the batch job scheduling method provided in this embodiment can generally be executed by device 103. Correspondingly, the batch job scheduling device provided in this embodiment can generally be installed in device 103. The batch job scheduling method provided in this embodiment can also be executed by a server or server cluster that is different from device 103 and capable of communicating with business systems 101, 102, and / or device 103. Correspondingly, the batch job scheduling device provided in this embodiment can also be installed in a server or server cluster that is different from device 103 and capable of communicating with business systems 101, 102, and / or device 103.

[0051] It should be understood that Figure 1 The number of business systems and devices shown is merely illustrative. Depending on implementation needs, any number of business systems and devices can be included.

[0052] The following will be based on Figure 1 The described scene, through Figures 2-5 The batch job scheduling method of the disclosed embodiments is described in detail.

[0053] Figure 2 A flowchart illustrating a batch job scheduling method according to an embodiment of the present disclosure is shown schematically.

[0054] like Figure 2 As shown, the batch job scheduling method of this embodiment can be applied to a job processing system, which includes a coordination service module and at least one business system. This batch job scheduling method can be executed by a device integrating a coordination service module (hereinafter referred to as a device). The batch job scheduling method of this embodiment may include operations S210 to S230.

[0055] In operation S210, task information related to each business system is obtained; the task information includes the batch job identifier of the batch job to which the task belongs, the task identifier of the task, the dependencies between tasks, and the service instance identifier of the service instance.

[0056] For example, a business system might process batch jobs, with each system utilizing at least one service instance to execute multiple tasks within the batch job. Task information can be configured during development in code or configuration files, and the identifiers of other tasks a task depends on can be used to define dependencies between tasks. These identifiers can be names or strings of numbers; meaningful names are acceptable. The current task doesn't need to know which business system its dependent tasks reside in or how to access them; it only needs to know the identifiers of its dependent tasks.

[0057] For example, when each business system starts up, it can trigger its own scanning logic to scan tasks with predefined tags and annotations. During the scanning process, it obtains task information and then sends a registration request to the device. The registration request can carry the aforementioned task information.

[0058] It should be understood that a batch job can be executed by multiple business systems, and each business system can send its own task information. Therefore, the device can obtain the task information related to each business system.

[0059] In operation S220, a directory tree is created based on the task information. The directory tree includes batch job nodes corresponding to batch job identifiers, task nodes corresponding to task identifiers, and service instance nodes corresponding to service instance identifiers. The values ​​of the task nodes are related to their dependencies.

[0060] For example, a directory tree can include nodes of multiple categories, and nodes of the same category can have child nodes, thus forming a tree structure. For instance, the node corresponding to a batch task can be used as the root node, and the nodes corresponding to the tasks included in the batch task can be used as child nodes of the root node.

[0061] In operation S230, task nodes are scheduled based on the directory tree to execute tasks in batch jobs.

[0062] For example, the directory tree includes a first node corresponding to the first task and a second node corresponding to the second task, with the second task depending on the first task. Because the directory tree can represent the dependencies between tasks, the second task node can be scheduled once the completion of the first task is confirmed.

[0063] According to the technical solution provided in this disclosure, as the business system starts, the node corresponding to the task can register itself. After the device obtains the task information, it can write the task information into a predetermined path in the directory tree according to the storage organization structure of the directory tree, thereby forming information describing the DAG in the directory tree. When a task changes, such as adding a task or modifying the dependencies between tasks, the newly added or modified task can also register itself, thereby adding a task node corresponding to the task in the directory tree. Compared with the prior art, the method provided in this disclosure does not require manual organization and maintenance of the DAG corresponding to the batch job using a batch job maintenance center, thereby reducing the workload of staff and improving accuracy.

[0064] In addition, when Zookeeper is used as the coordination service module, the registration information is stored in a distributed Zookeeper cluster, which also has the inherent characteristics of being distributed and highly available.

[0065] According to another embodiment of this disclosure, after creating the directory tree, the above method may further include the following operations: obtaining the status information of the service instance corresponding to the service instance node in the directory tree, and then deleting the node related to the service instance from the directory tree when it is determined that the service instance is in a shutdown state.

[0066] For example, a service instance node is a temporary node in the directory tree. The service instance maintains a connection with the device, such as performing heartbeat checks. When a service instance stops, it disconnects from the device. At this time, the device receives empty status information, thus determining that the service instance is in a stopped state. The device then deletes the nodes associated with the service instance from the directory tree. These associated nodes can include the service instance node itself.

[0067] In some embodiments, nodes associated with service instances can also be task nodes. For example, a business system uses service instances to perform tasks, so the status information of the service instances can reflect whether the business system is functioning correctly. When all service instances associated with the business system are down, it indicates that the business system is offline, and at this time, the task nodes corresponding to the tasks performed by the business system can be deleted.

[0068] In related technologies, when tasks change, such as reducing the number of tasks, staff need to manually synchronize and maintain the DAG.

[0069] According to the technical solution provided in this disclosure, when the business system is shut down, the task nodes become unavailable and will be automatically deleted from the directory tree, without manual intervention. This automatically updates the directory tree, ensuring that the task information in the directory tree is synchronized with the currently running tasks, thus achieving the effect of dynamically forming a DAG.

[0070] According to another embodiment of this disclosure, when service instances and tasks are respectively associated with the same business system, the service instance node corresponding to the service instance is a child node of the task node corresponding to the task. For example, a business system uses 10 service instances to form a cluster to provide services to the outside world, and the business system can execute 100 tasks. Then the directory tree includes 100 task nodes, and each task node includes 10 child nodes, and each of the 10 child nodes under each task node corresponds to a service instance.

[0071] Accordingly, the operation of deleting nodes related to service instances from the directory tree can include the following steps: deleting the service instance node corresponding to the service instance from the directory tree, and then determining whether the number of service instance nodes that satisfy the parent-child relationship with the task node is greater than 0. If the number is greater than 0, the task node is retained; if the number is equal to 0, the task node is deleted.

[0072] According to the technical solution provided in this disclosure, since the task node is determined based on the service instance node, it is possible to accurately determine whether the business system used to perform the task is offline, thereby updating the directory tree in a timely and accurate manner.

[0073] Figure 3 A schematic diagram of a directory tree according to an embodiment of the present disclosure is shown.

[0074] According to another embodiment of this disclosure, the operation of creating a directory tree based on task information may include the following operations: creating a service instance node 304 based on the service instance identifier of a service instance related to the business system; creating a parent node of service instance node 304 based on the task identifier of the task executed by the business system, resulting in task node 302, and determining the task identifiers of other tasks that the task depends on as the value 303 of task node 302; and creating a parent node of task node 302 based on the batch job identifier of the batch job to which the task belongs, resulting in batch job node 301.

[0075] The storage organization structure of task information in the device is as follows: Figure 3As shown. For example, / DAG is a fixed path where multiple batch jobs can be registered. / dag_name represents batch job node 301, which can use the batch job identifier as its node name. Different batch jobs correspond to different batch job identifiers. " / task node" represents task node 302, which can use the task identifier as its node name. The identifiers of other tasks that the task depends on can be used as the value 303 of task node 302, and the value 303 can be a comma-separated string. For example, if task D depends on tasks E and F, and task E depends on task G, then the value of the task node corresponding to task D is "the identifier of task E, the identifier of task F". The next level path of task node 302 is a temporary node, which can use the service instance identifier as its node name. The service instance identifier can include the IP address and port number of the business system associated with the service instance, which can uniquely identify a service instance. In addition, a business system can launch multiple service instances to form a cluster to provide services to the outside world. Providing services can mean executing multiple tasks related to the business system. For example, a business system can use 38 service instances to provide services to the outside world. The business system can execute 100 tasks. Correspondingly, the business system has 100 task nodes 302, and each task node 302 has 38 service instance nodes 304 at the next level.

[0076] It should be understood that in practical applications, service instance node 304 is a temporary node. When a service instance is shut down, the service instance node 304 corresponding to that service instance can be deleted from the directory tree. When all service instance nodes 304 under a certain " / task node" path have been deleted, it indicates that the business system corresponding to that task node 302 has been taken offline, and that task node 302 can be deleted from the directory tree.

[0077] According to the technical solution provided in the embodiments of this disclosure, the directory tree of the above-mentioned storage organization structure can clearly describe the hierarchical relationship of each node, and the DAG graph information can be generated relatively easily using the directory tree.

[0078] According to another embodiment of this disclosure, after obtaining the directory tree, a DAG can be generated based on the directory tree. The DAG includes nodes and edges. The nodes in the DAG correspond one-to-one with the task nodes in the directory tree, and the edges in the DAG indicate the dependencies between nodes.

[0079] For example, once the business system starts up, complete information about each DAG will be formed under the / DAG path or other predetermined paths. For each " / task node" path, its value represents the dependent tasks required before the task node can be executed, thus forming a complete DAG. For example, task nodes in the directory tree can be read to obtain the nodes in the DAG. Furthermore, if the identifiers of other tasks that a task depends on are stored as the values ​​of task nodes in the directory tree, the values ​​of each task node in the directory tree can be read to obtain the task dependencies, and the edges in the DAG can be determined based on the dependencies.

[0080] This embodiment generates a DAG based on a directory tree, and can then output the DAG for display, thereby facilitating staff to understand information related to batch operations.

[0081] In related technologies, the batch job maintenance center is responsible for not only maintaining the Directed Acyclic Graph (DAG) but also scheduling various tasks. The batch job maintenance center needs to record the execution status of each task, periodically retrieve and update the execution status of each task, calculate whether the next task satisfying the dependency relationship needs to be executed, and trigger its execution via RPC call when necessary. For example, for tasks D, E, and F mentioned above, the batch job maintenance center needs to periodically retrieve the execution status of tasks E and F, and trigger task D to execute after both have completed.

[0082] However, with the above technical solution, on the one hand, when the batch operation maintenance center is unavailable, the execution status of all tasks will be lost, and thus the task scheduling function cannot be realized.

[0083] On another front, the batch job maintenance center needs to periodically obtain the execution status of tasks before it can schedule tasks based on their dependencies. However, too large a polling interval will increase latency, while too small an interval will cause inflated CPU usage, wasting computing resources.

[0084] Please refer to Figure 4 and Figure 5 To address the aforementioned issues, in another embodiment of this disclosure, the batch job includes multiple task groups, each task group comprising a first task and a second task dependent on the first task. Scheduling task nodes based on a directory tree includes: for each task group, performing the following operations: obtaining the execution status of the first task, and then updating the value of a first status node in the directory tree according to the execution status, where the first status node is a child node of the first task node corresponding to the first task. Then, if the value of the first status node changes, triggering a listener registered by the second task, so that the second task can use the listener to determine the value of the first status node, and starting execution of the second task only after all tasks dependent on the second task have been successfully executed.

[0085] Figure 4 A schematic flowchart illustrating the execution status of a registration update task according to an embodiment of the present disclosure is shown.

[0086] For example, when a task is triggered for execution, its execution status needs to be registered and updated before the actual business logic processing begins. Figure 4 As shown, in this embodiment, the method for registering and updating the execution status of the task includes operations S401 to S405.

[0087] When operating S401, at the start of a task, the state node corresponding to the task can be initialized first. For example, when a task is executed, the task will send state information indicating the start of execution to the device. The device initializes the state node corresponding to the task (i.e., the / state path under the task node). The initialization process may include first checking whether the state node exists. If it does not exist, the path is created. If it already exists, its state value is cleared.

[0088] In operation S402, the value of the status node is updated to "processing". For example, during the execution of a task, the task sends status information indicating that it is being executed to the device, and the device updates the status value to "processing".

[0089] In operation S403, the task is executed, and its success is determined. If successful, operation S404 is executed; otherwise, operation S405 is executed. For example, after the task performs the actual business logic processing, it is also determined whether any exceptions have occurred in the business logic.

[0090] In operation S404, the value of the status node is updated to success. For example, if the business logic is detected to have executed successfully, a status message indicating successful execution is sent to the device, and the device updates the status value of the status node to success.

[0091] In operation S405, the value of the status node is updated to failure. For example, if a business logic execution failure is detected, status information indicating the failure is sent to the device, and the device updates the status value of the status node to failure.

[0092] The task execution process is completed through the above operations S401 to S405.

[0093] Figure 5 A schematic diagram illustrating responsive scheduling according to an embodiment of the present disclosure is shown. The following is in conjunction with... Figure 5 The scheduling operation described above will be explained.

[0094] For example, a batch job may include task D, task E, and task F, where task D depends on task E and task F.

[0095] The directory tree includes a batch job node 501 corresponding to the batch job. The next level of batch job node 501 includes task node E 5021, task node F 5022, and task node D 5023. The next level of task node E 5021 has a status node 5031, the next level of task node F 5022 has a status node 5032, and the next level of task node D 5023 has a status node 5033. The status nodes 5031, 5032, and 5033 have status values ​​5041, 5042, and 5043, respectively, to represent the execution status of task E, task F, and task D.

[0096] Task D will register two listeners: one listener with the listening path " / task node E / state" and the other listener with the listening path " / task node F / state".

[0097] An update to the state value of a state node will trigger the listener's listening processing logic to begin execution. For example, the device detects whether the state value of state node E in the directory tree has changed. If it has changed, it sends a message to task D, which depends on this task, including the state value of state node E. After receiving the message, task D triggers the listener's listening processing logic.

[0098] The listener's processing logic checks if the status value is successful. If it's not successful, it exits directly. For example, it checks if the status value is successful and exits if it's not. Taking tasks E, F, and D as an example, task D listens to the execution status of tasks E and F. A change in the execution status of task E triggers a check in task D, and a change in the execution status of task F triggers a check in task D. If task D determines that the status value of task E is not successful after checking its execution status, it doesn't need to check the execution status of task F and can exit directly without checking the execution status of all dependent tasks.

[0099] Furthermore, task D checks through a listener whether all its dependent tasks have succeeded. If all dependent nodes have executed successfully, it triggers the execution of its own task; otherwise, it continues to wait for other dependent tasks to trigger it, thus completing the entire process of triggering and executing dependent tasks. For example, once task D determines through the listener that tasks E and F have both completed, it will automatically execute its task processing logic.

[0100] As can be seen, using the above technical solution, the listening handler for task D is in a dormant state. When the state of task E changes, it will trigger task D to start its listening handler. That is, the task adopts a reactive triggering method; after the state of task E changes, task D responds to task E. If the states of both task E and task F do not change, the listening handler for task D will not be triggered.

[0101] The technical solution provided in this disclosure adopts a decentralized scheduling method, where task scheduling has distributed characteristics and does not rely on the triggering of a centralized node. Furthermore, a task determines whether it needs to execute based on the execution status of other tasks it depends on. That is, when other tasks a task depends on execute successfully, the task can sense the change in real time and trigger itself, achieving reactive task scheduling. Task nodes do not require central triggering, avoiding the problem of task execution failure or loss of task execution status due to batch job maintenance center downtime. Reactive scheduling has high real-time performance and avoids the resource waste and latency caused by periodic polling. Moreover, since the listener only focuses on the execution status of other tasks a task depends on, the successful execution of a task will only precisely affect other tasks directly dependent on it, thus avoiding large-scale event broadcasting.

[0102] The job processing system provided in this embodiment includes at least one business system and a coordination service module.

[0103] Each business system uses at least one service instance to execute multiple tasks in a batch job. Each business system scans its own multiple tasks at startup to obtain task information and send the task information.

[0104] The coordination service module is used to execute the methods described above. The coordination service module can use ZooKeeper, and the coordination service module and tasks can be integrated on different devices.

[0105] According to another embodiment of this disclosure, the batch job includes multiple task groups, each task group including a first task and a second task that depends on the first task. Accordingly, the business system is further configured to register a listener for monitoring a first status node with the coordination service module via the first task. The coordination service module is further configured to obtain the execution status of the first task; update the value of the first status node in the directory tree based on the execution status, where the first status node is a child node of the first task node corresponding to the first task; and trigger the listener registered by the second task if the value of the first status node changes. The business system is further configured to respond, for example, in response to the listener being triggered, determine the value of the status node using the listener, and begin executing the second task if all tasks dependent on the second task have been executed successfully.

[0106] Based on the above-described batch job scheduling method, this disclosure also provides a batch job scheduling apparatus. The following will be combined with... Figure 6 The device is described in detail.

[0107] Figure 6 A schematic block diagram of a batch job scheduling apparatus according to an embodiment of the present disclosure is shown.

[0108] like Figure 6 As shown, the batch job scheduling device 600 of this embodiment is applied to a job processing system. The job processing system includes a coordination service module and at least one business system for processing batch jobs. Each business system utilizes at least one service instance to execute multiple tasks in the batch job. The device 600 includes an information acquisition module 610, a creation module 620, and a scheduling module 630.

[0109] The information acquisition module 610 is used to acquire task information related to each business system; the task information includes the batch job identifier of the batch job to which the task belongs, the task identifier of the task, the dependencies between tasks, and the service instance identifier of the service instance. In one embodiment, the information acquisition module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0110] The creation module 620 is used to create a directory tree based on the task information. The directory tree includes batch job nodes corresponding to batch job identifiers, task nodes corresponding to task identifiers, and service instance nodes corresponding to service instance identifiers, and the values ​​of the task nodes are related to dependencies. In one embodiment, the creation module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0111] The scheduling module 630 is used to schedule task nodes based on the directory tree to execute tasks in a batch job. In one embodiment, the scheduling module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0112] According to another embodiment of this disclosure, the creation module includes a first creation submodule, a second creation submodule, and a third creation submodule. The first creation submodule is used to create a service instance node based on the service instance identifier of a service instance related to the business system. The second creation submodule is used to create a parent node of the service instance node based on the task identifier of the task executed by the business system, thus obtaining a task node; and to determine the task identifiers of other tasks that the task depends on as the value of the task node. The third creation submodule is used to create a parent node of the task node based on the batch job identifier of the batch job to which the task belongs, thus obtaining a batch job node.

[0113] According to another embodiment of this disclosure, the apparatus further includes a status acquisition module and a deletion module. The status acquisition module is used to acquire the status information of the service instance corresponding to the service instance node in the directory tree after the directory tree is created. The deletion module is used to delete the node related to the service instance from the directory tree if it is determined that the service instance is in a stopped state.

[0114] According to another embodiment of this disclosure, when the service instance and the task are respectively associated with the same business system, the service instance node corresponding to the service instance is a child node of the task node corresponding to the task; the deletion module includes a first deletion submodule and a second deletion submodule. The first deletion submodule is used to delete the service instance node corresponding to the service instance from the directory tree. The second deletion submodule is used to delete the task node when it is determined that the number of service instance nodes that satisfy the parent-child relationship with the task node is 0.

[0115] According to another embodiment of this disclosure, the above-mentioned apparatus further includes a generation module, which is used to generate directed acyclic graph information based on the directory tree; wherein the directed acyclic graph information includes nodes and edges, the nodes in the directed acyclic graph information correspond one-to-one with the task nodes in the directory tree, and the edges in the directed acyclic graph information indicate the dependencies between nodes.

[0116] According to another embodiment of this disclosure, the batch job includes multiple task groups, each task group including a first task and a second task that depends on the first task; the tuning module is used to perform the following operations for each task group: obtain the execution status of the first task; update the value of the first status node in the directory tree according to the execution status, the first status node being a child node of the first task node corresponding to the first task; and, if the value of the first status node changes, trigger a listener registered by the second task so that the second task can use the listener to determine the value of the first status node and start executing the second task if all tasks that the second task depends on have been executed successfully.

[0117] According to embodiments of this disclosure, any plurality of modules among the information acquisition module 610, creation module 620, and model adjustment module 630 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the information acquisition module 610, creation module 620, and model adjustment module 630 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the information acquisition module 610, creation module 620, and model adjustment module 630 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0118] Figure 7 A block diagram of an electronic device suitable for implementing a batch job scheduling method according to an embodiment of the present disclosure is shown schematically.

[0119] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0120] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0121] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0122] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0123] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0124] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the item recommendation method provided in the embodiments of this disclosure.

[0125] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0126] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0127] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0128] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0130] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0131] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A batch job scheduling method, applied to a job processing system, the job processing system including a coordination service module and at least one business system for processing batch jobs, each business system utilizing at least one service instance to execute multiple tasks in the batch job; the method includes: Obtain task information related to each of the aforementioned business systems; The task information includes the batch job identifier of the batch job to which the task belongs, the task identifier of the task, the dependencies between tasks, and the service instance identifier of the service instance. Based on the task information, a directory tree is created. The directory tree includes a batch job node corresponding to the batch job identifier, a task node corresponding to the task identifier, and a service instance node corresponding to the service instance identifier. The value of the task node is related to the dependency relationship. as well as Based on the directory tree, the task nodes are scheduled to execute tasks in the batch job; The step of creating a directory tree based on the task information includes: A service instance node is created based on the service instance identifier of the service instance associated with the business system; wherein, the service instance node serves as a temporary node in the directory tree, and the service instance identifier includes the IP address and port number of the business system associated with the service instance; Based on the task identifier of the task executed by the business system, a parent node is created for the service instance node, thus obtaining the task node; and the task identifiers of other tasks that the task depends on are determined as the value of the task node; and Based on the batch job identifier of the batch job to which the task belongs, create the parent node of the task node to obtain the batch job node.

2. The method according to claim 1, further comprising: After creating the directory tree, Obtain the status information of the service instance corresponding to the service instance node in the directory tree; as well as If it is determined that the service instance is in an offline state, the node associated with the service instance is deleted from the directory tree.

3. The method according to claim 2, wherein, When a service instance and a task are both associated with the same business system, the service instance node corresponding to the service instance is a child node of the task node corresponding to the task. Deleting the node associated with the service instance from the directory tree includes: Delete the service instance node corresponding to the service instance from the directory tree; as well as If the number of service instance nodes that satisfy the parent-child relationship with the task node is 0, then delete the task node.

4. The method according to claim 1, further comprising: Based on the directory tree, generate directed acyclic graph information; The directed acyclic graph information includes nodes and edges. The nodes in the directed acyclic graph information correspond one-to-one with the task nodes in the directory tree, and the edges in the directed acyclic graph information indicate the dependencies between nodes.

5. The method according to any one of claims 1 to 4, wherein the batch job comprises multiple task groups, each task group comprising a first task and a second task dependent on the first task; and the scheduling of the task nodes based on the directory tree comprises: For each task group, perform the following operations: Obtain the execution status of the first task; Based on the execution status, update the value of the first status node in the directory tree, where the first status node is a child node of the first task node corresponding to the first task; as well as If the value of the first state node changes, the listener registered by the second task is triggered so that the second task can use the listener to determine the value of the first state node and start executing the second task if all tasks on which the second task depends have been executed successfully.

6. A job processing system, comprising: At least one business system, each business system using at least one service instance to execute multiple tasks in a batch job, each business system being used to scan its own multiple tasks at startup, obtain task information, and send the task information; as well as A coordination service module for performing the method according to any one of claims 1 to 4.

7. The system according to claim 6, wherein, The batch job includes multiple task groups, and each task group includes a first task and a second task that depends on the first task; The coordination service module is further configured to: obtain the execution status of the first task; update the value of the first status node in the directory tree according to the execution status, wherein the first status node is a child node of the first task node corresponding to the first task; and trigger the listener registered by the second task when the value of the first status node changes. The business system is also configured to: register a listener for listening to the first state node with the coordination service module through the first task; in response to the listener being triggered, determine the value of the state node using the listener, and start executing the second task if all tasks on which the second task depends have been executed successfully.

8. A batch job scheduling apparatus, applied to a job processing system, the job processing system including a coordination service module and at least one business system for processing batch jobs, each business system utilizing at least one service instance to execute multiple tasks in the batch job; the apparatus includes: The acquisition module is used to acquire task information for each of the business systems; The task information includes the batch job identifier of the batch job to which the task belongs, the task identifier of the task, the dependencies between tasks, and the service instance identifier of the service instance. A creation module is used to create a directory tree based on the task information. The directory tree includes batch job nodes, task nodes, and service instance nodes corresponding to the batch job identifier, the task identifier, and the service instance identifier, respectively, and the value of the task node is related to the dependency relationship. The scheduling module is used to schedule the task nodes based on the directory tree to execute tasks in the batch job. The creation module includes a first creation submodule, a second creation submodule, and a third creation submodule. The first creation submodule is used to create a service instance node based on the service instance identifier of the service instance related to the business system. The service instance node serves as a temporary node in the directory tree, and the service instance identifier includes the IP address and port number of the business system related to the service instance. The second creation submodule is used to create the parent node of the service instance node based on the task identifier of the task executed by the business system, thus obtaining the task node; and to determine the task identifier of other tasks that the task depends on as the value of the task node; the third creation submodule is used to create the parent node of the task node based on the batch job identifier of the batch job to which the task belongs, thus obtaining the batch job node.

9. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images