Space mission dependent scheduling method and system across subsystems

Abstract

The application provides a cross-subsystem space task dependency scheduling method and system, comprising: configuring event triggers on a workflow, attributing a plurality of event triggers to a scheduling topology template; obtaining all triggers attributed to the topology scheduling trigger after the topology scheduling trigger, parsing into a cross-space flow dependency topology relationship, and finding a starting point workflow for scheduling; using a message bus mechanism to realize upstream and downstream workflow dependency notification of the cross-workspace flow dependency topology relationship; establishing a distributed memory bus mechanism, abstracting the message bus, using an SPI mechanism to realize decoupling of a TSS platform and a message queuing service and corresponding consumption logic, and realizing a plug-in message queuing service. The application maximizes the horizontal expansion mechanism of distributed computing power when facing the growth of business volume, supports dynamically increasing cluster scheduling nodes to cope with business pressure, and automatically calculates and generates a topology dependency relationship based on an algorithm to solve complex arrangement and scheduling problems between task flows.

Description

Technical Field

[0001] This invention relates to the field of task scheduling technology, and more specifically, to a spatial task-dependent scheduling method and system across subsystems. Background Technology

[0002] Huawei Cloud DGC data development module supports event-triggered job execution, thus enabling cross-space job scheduling via DIS or MRSKafka as job dependency links. Figure 1 After Job1 in workspace 1 completes its execution, a message can be sent using the DIS Client or Kafka Client to trigger Job2. Job2 is configured with event-triggered scheduling, and the job runs based on the message sent by the DIS Client or Kafka Client. Compared with Huawei Cloud's solution, the cross-space workflow dependency scheduling scheme of this invention is seamless in terms of sending and consuming dependency event messages, without requiring any additional steps or workflows. The workflow triggering of the topology and workspaces is relatively independent, resulting in lower coupling.

[0003] Patent document CN114510235A (application number: CN202210114410.3) discloses a full lifecycle management system and method for scientific computing programs. The system includes a construction environment subsystem and a production environment subsystem. The former provides computer resources for the construction process in the lifecycle of the scientific computing program, and the latter provides computer resources for the testing and deployment processes in the full lifecycle of the scientific computing program.

[0004] Alibaba Cloud DataWorks supports cross-workspace dependencies within the same region. Based on the principle of scheduling dependencies, it establishes node dependencies by using the output of an upstream node as the input of a downstream node, thus achieving cross-workspace scheduling dependencies. For example, adding the output of node A in workspace A as the input of node B in workspace B enables cross-workspace dependencies. The configuration method is the same as for general scheduling dependencies; detailed instructions can be found in Configuring Same-Period Scheduling Dependencies. Compared to Alibaba Cloud's solution, this invention's cross-workspace workflow dependency scheduling scheme supports a single template with multiple instances, offering a higher level of abstraction for real-world business scenarios, greater user flexibility, and the ability to reuse existing topology templates. Combined with dynamic parameter passing, it addresses the issue of creating numerous similar topologies or tasks. Furthermore, the topology trigger instances are business-isolated, supporting high-concurrency scheduling of multiple topology instances from a single topology template. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a space task-dependent scheduling method and system that spans subsystems.

[0006] The cross-subsystem space task dependency scheduling method provided by the present invention includes:

[0007] Step 1: Based on the TSS distributed workflow scheduling platform, configure event triggers on the workflow in the workspace, configure the corresponding upstream dependent workflows and the corresponding workflow running status results in the triggers, and classify the event triggers of multiple workflows into a scheduling topology template.

[0008] Step 2: After the topology scheduling is triggered, obtain all the triggers belonging to it, and use a preset algorithm to parse the unidirectional upstream and downstream workflow dependencies of multiple triggers into a complete cross-space flow dependency topology relationship, and find the starting workflow for scheduling.

[0009] Step 3: Use a message bus mechanism in the scheduling service logic to implement upstream and downstream workflow dependency notifications across workspace flow dependency topologies. Integrate flow dependency message triggering and message consumption mechanisms in the scheduling service, and implement flow control based on the node backpressure fault tolerance mechanism of the TSS workflow scheduling platform.

[0010] Step 4: Based on different production environment business volume and resource allocation, establish a distributed memory bus mechanism, abstract the message bus and use the SPI mechanism to decouple the TSS platform from the message queuing service and the corresponding consumption logic, and realize plug-in message queuing service.

[0011] Preferably, based on the workflow event message triggering mechanism, after the upstream task is completed, the scheduling service makes a logical judgment and sends an event message to the message queue, and the downstream message consumption mechanism consumes the event messages in the message queue, realizing cross-workspace flow-dependent event push triggering.

[0012] Preferably, the asynchronous flow-dependent event trigger node crash exception handling logic is set up to calculate and restore the state of the memory bus before the crash based on the topology instance, the corresponding message instance records, and the status of the completed workflow instance.

[0013] Preferably, when a topology instance trigger encounters a triggering exception, if the triggering exception is determined to be an exception that is not due to resource overload or does not meet the logical rule level, the trigger will automatically perform corresponding exception handling according to the various optional exception strategies configured by the user; if the exception is caused by the temporary overload of scheduling platform resources, the asynchronous trigger message compensation mechanism will be notified through the message bus to perform polling asynchronous processing to ensure that the topology instance resumes normal scheduling after the cluster load decreases.

[0014] Preferably, based on the upstream and downstream workflow dependencies of unidirectional triggers, a preset algorithm is used to calculate all the starting and ending workflow nodes in the entire topology template. The unidirectional upstream and downstream workflow dependencies in the triggers are quantified into directions to obtain a complete directed graph. Then, based on Tremaux search combined with DFS backtracking algorithm, topology relationship loop detection and judgment are performed to ensure that after the user configures workflow event triggers, the algorithm automatically detects whether the current trigger will introduce workflow dependency trigger loops after being added to the topology template. If a loop is detected, all workflows and corresponding triggers on the loop are automatically fed back in the order of dependency.

[0015] The cross-subsystem space task-dependent scheduling system provided by the present invention includes:

[0016] Module M1: Based on the TSS distributed workflow scheduling platform, it configures event triggers on workflows in the workspace, and configures the corresponding upstream dependent workflows and the corresponding workflow running status results in the triggers, and assigns the event triggers of multiple workflows to a scheduling topology template.

[0017] Module M2: After topology scheduling is triggered, it obtains all triggers belonging to it, and uses a preset algorithm to parse the unidirectional upstream and downstream workflow dependencies of multiple triggers into a complete cross-space flow dependency topology relationship, and finds the starting workflow for scheduling.

[0018] Module M3: In the scheduling service logic, a message bus mechanism is used to implement upstream and downstream workflow dependency notifications across workspace flow dependency topologies. Flow dependency message triggering and message consumption mechanisms are integrated into the scheduling service. Flow control is implemented based on the node backpressure fault tolerance mechanism of the TSS workflow scheduling platform.

[0019] Module M4: Based on different production environment business volume and resource allocation, a distributed memory bus mechanism is established. The message bus is abstracted and the SPI mechanism is used to decouple the TSS platform from the message queuing service and the corresponding consumption logic, realizing pluggable message queuing service.

[0020] Preferably, based on the workflow event message triggering mechanism, after the upstream task is completed, the scheduling service makes a logical judgment and sends an event message to the message queue, and the downstream message consumption mechanism consumes the event messages in the message queue, realizing cross-workspace flow-dependent event push triggering.

[0021] Preferably, the asynchronous flow-dependent event trigger node crash exception handling logic is set up to calculate and restore the state of the memory bus before the crash based on the topology instance, the corresponding message instance records, and the status of the completed workflow instance.

[0022] Preferably, when a topology instance trigger encounters a triggering exception, if the triggering exception is determined to be an exception that is not due to resource overload or does not meet the logical rule level, the trigger will automatically perform corresponding exception handling according to the various optional exception strategies configured by the user; if the exception is caused by the temporary overload of scheduling platform resources, the asynchronous trigger message compensation mechanism will be notified through the message bus to perform polling asynchronous processing to ensure that the topology instance resumes normal scheduling after the cluster load decreases.

[0023] Preferably, based on the upstream and downstream workflow dependencies of unidirectional triggers, a preset algorithm is used to calculate all the starting and ending workflow nodes in the entire topology template. The unidirectional upstream and downstream workflow dependencies in the triggers are quantified into directions to obtain a complete directed graph. Then, based on Tremaux search combined with DFS backtracking algorithm, topology relationship loop detection and judgment are performed to ensure that after the user configures workflow event triggers, the algorithm automatically detects whether the current trigger will introduce workflow dependency trigger loops after being added to the topology template. If a loop is detected, all workflows and corresponding triggers on the loop are automatically fed back in the order of dependency.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] This invention uses a memory message bus in conjunction with message queues to achieve real-time notification of workflow-dependent events. This directly avoids accessing the storage layer, greatly improving scheduling and processing performance. Moreover, it maximizes the horizontal scaling mechanism of distributed computing power when facing business volume growth, supports dynamically adding cluster scheduling nodes to cope with business pressure, and thus perfectly solves the computing power bottleneck problem. Attached Figure Description

[0026] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0027] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0028] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0029] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0030] Example:

[0031] like Figure 2 This invention proposes a cross-subsystem spatial task dependency scheduling method, including: a cross-workspace workflow dependency triggering solution based on workflow templates under workspace and multi-instance mode; setting event triggers at the workflow granularity to decouple the relationship between topology and workflow; implementing cross-workspace workflow upstream and downstream dependency notification based on the TSS workflow scheduling platform, changing the message timed pull mode to an event-triggered push mode, solving problems such as real-time task consumption and storage layer access pressure caused by increasing task volume; centrally parsing the upstream and downstream workflow dependencies of all triggers belonging to the topology template, and finally parsing a complete workflow dependency topology relationship and finding the started workflow node; setting up a distributed memory message bus service, in conjunction with the TSS workflow scheduling platform, to achieve memory-level MQ service to save physical resources; and setting up a pluggable cross-workspace flow dependency message bus mechanism to adapt to third-party MQ services, maximizing the availability and adaptability of the TSS workflow scheduling platform.

[0032] We develop workflow event triggers that allow users to configure upstream and downstream dependencies, triggering logic rules, timeout policies, and other features of the workflow according to their own business needs.

[0033] We developed a topology management system that assigns event triggers to topology templates. Through our self-developed algorithm, we uniformly calculate, parse, and ultimately sort out a complete workflow dependency trigger topology relationship.

[0034] Develop a workflow event message triggering mechanism. After the upstream task is completed, the scheduling service makes a logical judgment and sends an event message to the MQ. The downstream message consumption mechanism consumes the event messages in the MQ, realizing cross-workspace flow dependency event push triggering.

[0035] The development of TSS workflow scheduling tasks triggered by flow-dependent messages enables real-time resource control of cross-workspace flow-dependent scheduling tasks based on the resource groups and corresponding resource management mechanisms of the self-developed TSS workflow scheduling platform.

[0036] Develop logic for handling asynchronous stream-dependent event-triggered node crashes. Based on the topology instance, corresponding message instance records, and the status of completed workflow instances, calculate and restore the memory bus state before the crash.

[0037] The message bus is developed in a pluggable manner, using SPI to work with the abstract encapsulation of event message production and consumption interfaces, and the specific actual logic is pushed down to the specific subclass implementation, thereby decoupling the cross-workflow dependency triggering scheme from the MQ.

[0038] When a topology instance trigger encounters a triggering exception, if the triggering exception is determined to be an exception that is not due to resource overload or does not meet the logical rule level, the trigger will automatically perform corresponding exception handling according to the various optional exception strategies configured by the user; if the exception is caused by the temporary overload of scheduling platform resources, then the asynchronous trigger message compensation mechanism will be notified through the message bus to perform polling asynchronous processing to ensure that the topology instance resumes normal scheduling after the cluster load decreases.

[0039] First, based on the upstream and downstream workflow dependencies of the self-developed unidirectional triggers, the algorithm accurately calculates all the starting and ending workflow nodes in the entire topology template. After obtaining these nodes, the unidirectional upstream and downstream workflow dependencies in the triggers can be quantified into directions, ultimately resulting in a complete directed graph. After obtaining the topology directed graph, based on the Tremaux search combined with the DFS backtracking algorithm, a self-developed topology relationship loop detection and judgment algorithm is implemented. This ensures that after the user configures workflow event triggers, the algorithm automatically detects whether adding the current trigger to the topology template will introduce workflow dependency trigger loops. If a loop is detected, it will automatically provide feedback and prompts for all workflows and corresponding triggers on the loop according to their dependency order.

[0040] Currently, existing cross-workspace flow dependency products or solutions in the industry treat each cross-workspace flow dependency scheduling job as globally unique. For example, an instance of a dependent task can only be triggered once before the entire topology scheduling is complete, or the system checks whether the task has been triggered within a certain period. This design is inaccurate and, in high-frequency triggering scenarios of cross-workspace flow dependencies, it cannot guarantee a one-to-one mapping between the executing workflow and the cross-workspace flow dependency scheduling job, leading to serious problems such as data corruption and scheduling anomalies. Furthermore, most mainstream cross-workspace flow dependency products or solutions currently use fixed-interval polling to monitor and determine whether the dependent workflow status has changed. If the polling interval is shortened, it results in significant I / O waste at the storage layer; if the polling interval is extended, it introduces latency issues for the scheduling task dependency triggering. Moreover, the query pressure on the underlying layer increases linearly with the number of scheduling tasks. Therefore, as the business volume increases in the later stages, the performance and stability of the fixed-interval polling solution become bottlenecks. Therefore, we have developed and designed an event bus notification mechanism to generate corresponding notification messages based on the changes in the running state of each workflow instance. We use a memory message bus in conjunction with MQ to realize real-time notification of workflow-dependent events. This directly avoids accessing the storage layer, greatly improves scheduling and processing performance, and maximizes the horizontal scaling mechanism of distributed computing power when facing business volume growth. It supports dynamically adding cluster scheduling nodes to cope with business pressure, thus perfectly solving the computing power bottleneck problem.

[0041] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0042] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A cross-subsystem spatial task dependency scheduling method, characterized in that, Includes the following steps: Step 1: Based on the TSS distributed workflow scheduling platform, configure event triggers on the workflow in the workspace, configure the corresponding upstream dependent workflows and the corresponding workflow running status results in the triggers, and classify the event triggers of multiple workflows into a scheduling topology template. Step 2: After the topology scheduling is triggered, obtain all the triggers belonging to it, and use a preset algorithm to parse the unidirectional upstream and downstream workflow dependencies of multiple triggers into a complete cross-space flow dependency topology relationship, and find the starting workflow for scheduling. Step 3: Use a message bus mechanism in the scheduling service logic to implement upstream and downstream workflow dependency notifications across workspace flow dependency topologies. Integrate flow dependency message triggering and message consumption mechanisms in the scheduling service, and implement flow control based on the node backpressure fault tolerance mechanism of the TSS workflow scheduling platform. Step 4: Based on different production environment business volume and resource allocation, establish a distributed memory bus mechanism, abstract the message bus and use the SPI mechanism to decouple the TSS platform from the message queuing service and the corresponding consumption logic, and realize plug-in message queuing service.

2. The cross-subsystem spatial task dependency scheduling method according to claim 1, characterized in that, Based on the workflow event message triggering mechanism, after the upstream task is completed, the scheduling service makes a logical judgment and sends an event message to the message queue. The downstream message consumption mechanism consumes the event messages in the message queue, realizing cross-workspace flow-dependent event push triggering.

3. The cross-subsystem spatial task dependency scheduling method according to claim 1, characterized in that, Configure asynchronous flow dependency event trigger node crash exception handling logic, calculate and restore the memory bus state before the crash based on the topology instance, corresponding message instance records, and the status of completed workflow instances.

4. The cross-subsystem spatial task dependency scheduling method according to claim 1, characterized in that, When a trigger for a topology instance encounters a triggering exception, if the triggering exception is determined to be an exception that is not due to resource overload or does not meet the logical rule level, the trigger will automatically perform the corresponding exception handling according to the various optional exception strategies configured by the user. If the anomaly is caused by temporary overload of scheduling platform resources, the asynchronous trigger message compensation mechanism is notified through the message bus to perform asynchronous polling to ensure that the topology instance resumes normal scheduling after the cluster load decreases.

5. The cross-subsystem spatial task dependency scheduling method according to claim 1, characterized in that, Based on the upstream and downstream workflow dependencies of unidirectional triggers, a preset algorithm is used to calculate all the starting and ending workflow nodes in the entire topology template. The unidirectional upstream and downstream workflow dependencies in the triggers are quantified as directions to obtain a complete directed graph. Then, based on Tremaux search combined with DFS backtracking algorithm, topology relationship loop detection and judgment are performed to ensure that after the user configures workflow event triggers, the algorithm automatically detects whether the current trigger will introduce workflow dependency trigger loops after being added to the topology template. If a loop is detected, all workflows and corresponding triggers on the loop are automatically fed back in the order of dependency.

6. A cross-subsystem space task-dependent scheduling system, characterized in that, Includes the following modules: Module M1: Based on the TSS distributed workflow scheduling platform, it configures event triggers on workflows in the workspace, and configures the corresponding upstream dependent workflows and the corresponding workflow running status results in the triggers, and assigns the event triggers of multiple workflows to a scheduling topology template. Module M2: After topology scheduling is triggered, it obtains all triggers belonging to it, and uses a preset algorithm to parse the unidirectional upstream and downstream workflow dependencies of multiple triggers into a complete cross-space flow dependency topology relationship, and finds the starting workflow for scheduling. Module M3: In the scheduling service logic, a message bus mechanism is used to implement upstream and downstream workflow dependency notifications across workspace flow dependency topologies. Flow dependency message triggering and message consumption mechanisms are integrated into the scheduling service. Flow control is implemented based on the node backpressure fault tolerance mechanism of the TSS workflow scheduling platform. Module M4: Based on different production environment business volume and resource allocation, a distributed memory bus mechanism is established. The message bus is abstracted and the SPI mechanism is used to decouple the TSS platform from the message queuing service and the corresponding consumption logic, realizing pluggable message queuing service.

7. The cross-subsystem space task-dependent scheduling system according to claim 6, characterized in that, Based on the workflow event message triggering mechanism, after the upstream task is completed, the scheduling service makes a logical judgment and sends an event message to the message queue. The downstream message consumption mechanism consumes the event messages in the message queue, realizing cross-workspace flow-dependent event push triggering.

8. The cross-subsystem space task-dependent scheduling system according to claim 6, characterized in that, Configure asynchronous flow dependency event trigger node crash exception handling logic, calculate and restore the memory bus state before the crash based on the topology instance, corresponding message instance records, and the status of completed workflow instances.

9. The cross-subsystem space task-dependent scheduling system according to claim 6, characterized in that, When a trigger for a topology instance encounters a triggering exception, if the triggering exception is determined to be an exception that is not due to resource overload or does not meet the logical rule level, the trigger will automatically perform the corresponding exception handling according to the various optional exception strategies configured by the user. If the anomaly is caused by temporary overload of scheduling platform resources, the asynchronous trigger message compensation mechanism is notified through the message bus to perform asynchronous polling to ensure that the topology instance resumes normal scheduling after the cluster load decreases.

10. The cross-subsystem space task-dependent scheduling system according to claim 6, characterized in that, Based on the upstream and downstream workflow dependencies of unidirectional triggers, a preset algorithm is used to calculate all the starting and ending workflow nodes in the entire topology template. The unidirectional upstream and downstream workflow dependencies in the triggers are quantified as directions to obtain a complete directed graph. Then, based on Tremaux search combined with DFS backtracking algorithm, topology relationship loop detection and judgment are performed to ensure that after the user configures workflow event triggers, the algorithm automatically detects whether the current trigger will introduce workflow dependency trigger loops after being added to the topology template. If a loop is detected, all workflows and corresponding triggers on the loop are automatically fed back in the order of dependency.

Images