Distributed DAG batch scheduling method and device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses a distributed DAG batch scheduling method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring flow batch front tasks and distributing the flow batch front tasks to any server in an actuator cluster; when the execution result is successful execution, fragment task information returned by the server is received; creating a plurality of fragmentation tasks according to the fragmentation task information; distributing the plurality of fragmented tasks to an actuator cluster for processing according to a set task issuing strategy; obtaining execution results of the plurality of fragmented tasks from an executor cluster; updating the state of the stream batch according to the plurality of execution results; and generating fragmented tasks of the next stream batch depending on the stream batch based on the plurality of execution results, and allocating the plurality of fragmented tasks to an actuator cluster for processing according to a set task issuing strategy. By implementing the method provided by the embodiment of the invention, the flexibility and the overall efficiency of the system can be improved, and the stability and the consistency are also ensured.

Description

Technical Field

[0001] The present invention relates to a batch job scheduling method, and more specifically to a distributed DAG batch scheduling method, apparatus, computer equipment, and storage medium. Background Art

[0002] XXL-Job and ES-Job are common large-scale data processing task scheduling platforms, primarily used for the management and scheduling of batch tasks. However, in actual use and construction, they still have some limitations that cannot fully meet business needs. Both platforms support group scheduling, individual scheduling, serial scheduling, and parallel scheduling of batch tasks, and provide batch-based sharding task expansion capabilities. However, their functionality still has some limitations when handling task sharding, particularly in terms of passing sharding information across batches. For example, they can only perform sharding operations and pass sharding information within the same batch job, but cannot pass sharding information across batches. Specifically, the execution of the next batch can only be triggered after all sharding tasks in the current batch are completed and successfully confirmed. However, during this process, the sharding information of the previous batch cannot be passed to the next batch.

[0003] Batch jobs typically schedule business tasks in batches, steps, and in a distributed manner. However, in real-world applications, demand fluctuates significantly, and some sharded tasks may need to be further split. This can lead to the next batch of tasks being split twice or more, based on the current sharding. However, XXL-Job and ES-Job do not support further task splitting during this process, resulting in a lack of flexibility in real-world applications and an inability to effectively address complex business needs.

[0004] Because these two platforms don't support cross-batch transfer of shard information and triggering subsequent shard tasks, they sometimes waste computing resources. For example, the preceding batch task must wait until all shard tasks are completed before triggering the subsequent batch execution, during which time the executor resources for the subsequent tasks may be idle. Furthermore, the execution time and resource consumption of tasks across shards are often inconsistent, meaning that the execution efficiency of the entire batch job is limited by the most time-consuming shard, slowing down the overall business execution efficiency, extending the critical path time, and wasting system resources.

[0005] In summary, although XXL-Job and ES-Job provide certain functions for batch job scheduling, they still have significant shortcomings in handling complex sharding tasks, cross-batch information transmission, and flexible scheduling.

[0006] Therefore, it is necessary to design a new method to improve the flexibility and overall efficiency of the system while also ensuring stability and consistency, and to solve the problems existing in existing technologies in handling complex sharding tasks, cross-batch information transmission, and flexible scheduling. Summary of the Invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a distributed DAG batch scheduling method, device, computer equipment and storage medium.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: a distributed DAG batch scheduling method, comprising:

[0009] Obtain the preceding tasks of the stream batch and assign them to any server in the executor cluster to obtain the execution results;

[0010] When the execution result is successful, receiving the sharding task information returned from the server;

[0011] Create several sharding tasks according to the sharding task information;

[0012] Allocate several of the shard tasks to the executor cluster for processing according to the set task delivery strategy;

[0013] Obtaining execution results of the plurality of shard tasks from the executor cluster;

[0014] updating the status of the flow batch according to a number of the execution results;

[0015] Based on the plurality of execution results, a shard task of the next stream batch that depends on the stream batch is generated, and the set task delivery strategy is executed to distribute the plurality of shard tasks to the executor cluster for processing.

[0016] A further technical solution is: the shard task information includes a number of shard tasks into which the task corresponding to the stream batch is divided and input parameters of each shard task.

[0017] Its further technical solution is: the task issuing strategy includes a random issuing strategy and a polling issuing strategy.

[0018] A further technical solution is: the rules for determining the input parameters of each shard task include:

[0019] When the next stream batch and the current stream batch are in the same group and the next stream batch is not split twice, the input parameter of each of the sharding tasks corresponding to the next stream batch is the execution result of the sharding task of the current stream batch.

[0020] A further technical solution is: the rules for determining the input parameters of each of the sharding tasks further include:

[0021] When the next stream batch and the current stream batch are in the same group and the next stream batch is split twice, the input parameters of all sharding tasks of the next stream batch after the second split are the execution results of the sharding tasks of the current stream batch.

[0022] A further technical solution thereof is: the updating of the state of the flow batch according to a plurality of the execution results includes:

[0023] When several execution results are all successful, and when there is a previous batch and the status of the previous batch is successful, the status of the current batch is set to successful.

[0024] A further technical solution is that the updating of the state of the flow batch according to the plurality of execution results includes:

[0025] When a shard task in a stream batch fails to execute, other shard tasks remain in the execution state. After all tasks are completed, the stream batch execution state is set to failure. For the next stream batch that depends on the previous stream batch, the state of the next stream batch is changed to failure after ensuring that all possible shard tasks are completed.

[0026] The present invention also provides a distributed DAG batch scheduling device, comprising:

[0027] The allocation unit is used to obtain the preceding tasks of the stream batch and allocate them to any server in the executor cluster to obtain the execution results;

[0028] A shard receiving unit, configured to receive shard task information returned from the server when the execution result is successful;

[0029] A creating unit, configured to create a plurality of sharding tasks according to the sharding task information;

[0030] A dispatching unit, configured to dispatch a plurality of the shard tasks to an executor cluster for processing according to a set task dispatching strategy;

[0031] A result acquisition unit, configured to acquire execution results of the plurality of shard tasks from the executor cluster;

[0032] an updating unit, configured to update the state of the flow batch according to a plurality of the execution results;

[0033] A new batch task generation unit is used to generate shard tasks of the next stream batch that depends on the stream batch based on the plurality of execution results, and to execute the set task delivery strategy to distribute the plurality of shard tasks to the executor cluster for processing.

[0034] The present invention further provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0035] The present invention also provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0036] The beneficial effects of the present invention compared with the prior art are as follows: the present invention flexibly distributes the preceding tasks of the stream batch to any server in the executor cluster for processing, and dynamically creates specific shard tasks based on the returned shard task information after successful execution. This process enhances the system's adaptability and task refinement capabilities; the shard tasks are efficiently distributed to different executors in the cluster for processing based on the set task delivery strategy. This method not only optimizes resource utilization, but also improves processing speed and load balancing; the execution results of all shard tasks are obtained and the status of the stream batch is updated accordingly, ensuring that even if a shard fails, the status of the entire stream batch can be accurately reflected, thereby maintaining the stability and consistency of the system; the shard tasks of the next stream batch are automatically generated based on the results of the current stream batch, and the effective task delivery strategy is continued to be applied, realizing automatic management of data flow and dependencies under complex business logic, and solving the problem of cross-batch information transmission; the overall system can not only handle complex shard tasks, but also flexibly respond to various scheduling requirements, significantly improving the response capability and processing efficiency when facing complex task scenarios, and overcoming the shortcomings of the prior art in this regard.

[0037] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A schematic diagram of an application scenario of the distributed DAG batch scheduling method provided by an embodiment of the present invention;

[0040] Figure 2A schematic diagram of the process of a distributed DAG batch scheduling method provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a flow batch provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of the flow batch execution process provided by an embodiment of the present invention;

[0043] Figure 5 A schematic block diagram of a distributed DAG batch scheduling device provided by an embodiment of the present invention;

[0044] Figure 6 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0046] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0047] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0048] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0049] See also Figure 1 and Figure 2 , Figure 1 Schematic diagram of an application scenario of the distributed DAG batch scheduling method provided by an embodiment of the present invention. Figure 2A schematic flow chart of the distributed DAG batch scheduling method provided in an embodiment of the present invention. The distributed DAG batch scheduling method is applied in a scheduler. The scheduler interacts with the executor cluster for data, and by dynamically allocating pre-tasks and sharding tasks to the executor cluster, it ensures efficient parallel processing while improving system flexibility. Based on the determination rules of the input parameters, especially for the information transmission between batches of the same grouped stream, the data consistency and continuity between cross-batch tasks are ensured, and the management problem of complex sharding tasks is solved. The use of diverse task issuance strategies such as random or polling enhances the adaptability and resource utilization efficiency of the system and meets the scheduling requirements in different scenarios. For tasks that fail to execute, the state is updated uniformly after waiting for all possible sharding tasks to complete, ensuring the overall stability of the system and the consistency of the data. Through the above mechanism, this method effectively improves the ability to handle complex sharding tasks, cross-batch information transmission and flexible scheduling, and overcomes the limitations of the existing technology.

[0050] Specifically, the scheduler is responsible for reading batch configurations from the database, generating task instances (including sharding logic), distributing tasks to the executor cluster, and tracking task status. The executor cluster receives tasks issued by the scheduler, executes them, and feeds the execution results back to the scheduler.

[0051] The minimum execution unit is a specific task implementation class corresponding to each task. Batch types include regular batches and stream batch groups. Regular batches are independent tasks or sharded task groups. Stream batch groups are a set of associated batches (explicit grouping is required) that can communicate sharding information, that is, several stream batches. Stream batches are processed using a Directed Acyclic Graph (DAG), where each node in the DAG represents a batch and dependencies are defined by edges. Subsequent batches can only be started if all preceding batches have executed successfully. Shard information (number of tasks, input / output) is completely independent between regular batches.

[0052] like Figure 3 As shown, stream batch groups require a group identifier, meaning they must be explicitly marked as belonging to the same group (e.g., group L1 in the example). The first stream batch in a group (e.g., BATCHL0001) must be configured with a predecessor task sharding mode, meaning its shard count is inherited from the sharding results of the preceding batch in the DAG. Subsequent stream batches (e.g., BATCHL0002) automatically inherit the sharding information of the previous stream batch in the group, forming a continuous shard delivery chain.

[0053] Figure 2 This is a flow chart of the distributed DAG batch scheduling method provided by an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S170.

[0054] S110 , obtaining the preceding tasks of the stream batch and assigning them to any server in the executor cluster to obtain the execution results.

[0055] In this embodiment, the execution result refers to whether the server can successfully execute the preceding task after it is assigned to the server.

[0056] Specifically, the scheduler is responsible for identifying and obtaining the prerequisite tasks required for a stream batch (e.g., BatchA) and assigning them to any server in the executor cluster for processing. This prerequisite task is an independent task that must be executed before the main task of BatchA. The task assignment process may involve selecting the most appropriate server based on factors such as the server's current load and performance. Once a task is assigned to a server, that server will begin executing the business logic of the prerequisite task.

[0057] The scheduler sends a command to a server in the executor cluster, instructing it to execute a specific precondition to determine whether the precondition can be successfully executed. After completing the precondition, the executor returns a result indicating success or failure to the scheduler. If the execution fails, the entire BatchA execution process ends because the precondition was not met. If the execution succeeds, the process proceeds to step S120.

[0058] S120: When the execution result is successful, receive the sharding task information returned from the server.

[0059] In this embodiment, the sharding task information includes the number of sharding tasks that the task corresponding to the stream batch is divided into and the input parameters of each sharding task. Specifically, a sharding task refers to the n subtasks (BatchA1, BatchA2, ..., BatchAn) that are divided into the original large task (such as BatchA). Each subtask is independent and can be executed separately. For each of the above subtasks, there is a set of specific input parameters. These parameters define the data range that the sharding task should process or other necessary operating conditions.

[0060] When the next stream batch and the current stream batch are in the same group and the next stream batch is not split twice, the input parameter of each of the sharding tasks corresponding to the next stream batch is the execution result of the sharding task of the current stream batch.

[0061] When the next stream batch and the current stream batch are in the same group and the next stream batch is split twice, the input parameters of all sharding tasks of the next stream batch after the second split are the execution results of the sharding tasks of the current stream batch.

[0062] This sharding mechanism allows for dynamic information transfer between tasks. For example, subsequent batches (e.g., BatchB) can adjust their task division and input parameters based on the actual execution output of the previous batch (e.g., BatchA). This eliminates the need to wait for all shards to complete before starting the next shard, significantly improving overall system efficiency. If a shard fails, the system has mechanisms in place to handle it. For example, a scanning thread checks the status of other shards and updates the overall status accordingly, ensuring system robustness and reliability.

[0063] In summary, the concept of sharded task information is a strategy used in distributed computing to achieve task parallelization, optimize resource utilization, and enhance system flexibility. In this way, complex batch jobs can be managed and executed more efficiently.

[0064] Specifically, after confirming the success of the predecessor task, the next steps involve processing the subsequent operations triggered by the predecessor task. Specifically, this includes receiving information from the server that executed the predecessor task about how to further split BatchA into multiple shard tasks, as well as the specific input parameters for each shard task.

[0065] Specifically, the server that successfully executes the pre-task generates shard task information, which includes the n shard tasks into which BatchA is divided and the input parameters for each shard task. This information is then sent back to the scheduler. After receiving the complete shard task information, the scheduler can use this data to plan and manage the execution order and resource allocation of BatchA's various shard tasks, ensuring the efficient and orderly completion of the entire batch task.

[0066] Through the above two steps, the system can effectively manage and optimize the execution process of complex tasks, ensuring high operational efficiency even in the face of complex dependencies and technical challenges.

[0067] S130: Create several sharding tasks according to the sharding task information.

[0068] In this example, the scheduler first saves the shard task information for BatchA to the database. This information includes, but is not limited to, each shard task's identifier, input parameters, and any other necessary metadata. This information serves as the basis for subsequently creating specific shard tasks and ensures that the system can track and manage the status of each shard task.

[0069] Based on the shard task information stored in the database, the scheduler generates n specific shard tasks for BatchA, such as BatchA1, BatchA2, ..., BatchAn. Each shard task has unique input parameters that define the data range or execution conditions it must process. For example, for a large data processing task, input parameters might specify the location and size of the data block to be processed; for a complex computation task, input parameters might be initial values ​​or configuration options.

[0070] S140: Allocate the plurality of shard tasks to an executor cluster for processing according to a set task delivery strategy.

[0071] In this embodiment, the task issuing strategy includes a random issuing strategy and a round-robin issuing strategy.

[0072] Once all the shard tasks have been created, the next step is to decide how to assign them to the executor cluster for processing. This step usually involves using some kind of task distribution strategy, such as random distribution, polling distribution, or other more complex strategies.

[0073] Random allocation means randomly selecting a server from the executor cluster to execute a shard task, while polling allocation assigns tasks to each server in the cluster in turn.

[0074] Adopting an appropriate task delivery strategy helps balance the load, avoid overloading certain servers, and maximize resource utilization.

[0075] Finally, based on the selected task delivery strategy, the scheduler delivers these sharded tasks to the executor cluster, which means that each sharded task will be sent to one or more servers in the cluster for execution.

[0076] After receiving the sharding task, the executor will execute the corresponding business logic based on its input parameters and return the result to the scheduler after completion.

[0077] Through this process, the scheduler not only effectively breaks down a large task into multiple smaller tasks that can be processed in parallel, but also ensures that these tasks are efficiently and evenly distributed across the executor cluster, thereby improving the processing power and efficiency of the entire system. This mechanism is particularly critical for processing large data sets or performing complex computations, as it allows the system to leverage the advantages of distributed computing to accelerate task completion time.

[0078] S150: Obtain execution results of several shard tasks from the executor cluster.

[0079] In this embodiment, the execution result refers to the result obtained by the executor cluster processing the shard task.

[0080] Specifically, when the executor completes a shard task, it transmits the task's execution results (including but not limited to output data and status information) back to the scheduler. The scheduler receives these execution results and associates them with the corresponding shard task for storage or further processing. The execution results of each shard task are independent and unique, and together they form part of the overall execution status of the current stream batch.

[0081] S160: Update the status of the flow batch according to the plurality of execution results.

[0082] In this embodiment, when several execution results are all successful, and when there is a previous flow batch and the status of the previous flow batch is successful, the current flow batch status is set to successful.

[0083] When a shard task in a stream batch fails to execute, other shard tasks remain in the execution state. After all tasks are completed, the stream batch execution state is set to failure. For the next stream batch that depends on the previous stream batch, the state of the next stream batch is changed to failure after ensuring that all possible shard tasks are completed.

[0084] Specifically, this phase focuses on updating the state of the current stream batch based on the execution results of multiple shard tasks obtained from the executor cluster.

[0085] Only when all shard tasks (BatchA1, BatchA2, ..., BatchAn) are successfully executed and return successful execution results, the status of the current stream batch is considered to be updated to "successful".

[0086] If a previous batch exists, you also need to check whether its status is also successful. This is because in some application scenarios, there may be dependencies between batches, and the successful execution of the previous batch is a prerequisite for the correct processing of subsequent batches.

[0087] Once the above two conditions are met, that is, all shard tasks are successfully executed and the status of the previous stream batch is also successful, the scheduler updates the status of the current stream batch to "success".

[0088] Specifically, if any shard task fails, the entire batch will be marked as failed, even if all other tasks succeed. This is because in a distributed system, failure at any link can prevent the entire process from completing as expected.

[0089] For tasks that have not yet completed execution, the system will continue to wait until they are all completed before making a decision. This is done to ensure that the state of the stream batch is not changed prematurely due to the delay of individual tasks.

[0090] If the next batch depends on the results of the current batch, after confirming that the current batch has failed, ensure that all possible sharding tasks have completed before changing the status of the next batch to failed. This helps prevent the propagation of incorrect status due to incomplete information.

[0091] This approach not only accurately reflects the actual execution status of each sharded task, but also effectively manages and controls the dependencies and state transitions between batches, thereby ensuring the stability and reliability of the entire operation chain. Furthermore, this provides detailed basic data support for subsequent data analysis and troubleshooting.

[0092] S170: Generate sharding tasks for the next stream batch that depends on the stream batch based on the plurality of execution results, and execute the set task dispatching strategy to distribute the plurality of sharding tasks to the executor cluster for processing.

[0093] When the loop condition of step S170 is exited, all stream batches are processed.

[0094] Specifically, the number of fragments of the first stream batch in the group = the final number of fragments of the preceding non-stream batch.

[0095] The number of shards in the next stream batch equals the number of shards in the previous stream batch within the group, and the task input can depend on the output of the previous stream (such as the shard data key value).

[0096] like Figure 4 As shown in the figure, if the number of shards of BATCHL0001 is 5, BATCHL0002 automatically generates 5 shard tasks and may receive the output of the corresponding shard of BATCHL0001 as input.

[0097] Streaming batches within a group must execute sequentially in the order in which they are grouped, and shard transfers cannot cross groups. All streaming batches within a group are considered a logical whole; all must succeed before subsequent non-streaming batches can be triggered. The executor cluster transmits task results (success / failure and output data) back to the scheduler. The scheduler updates the task status and, for streaming batch groups, synchronously maintains the context of the shards within the group. If any task fails, the associated batch is marked as failed, blocking subsequent dependent batches. Failure of a single shard within a streaming batch group can be retried or rolled back across the entire group, depending on the configuration.

[0098] Specifically, an executor is responsible for processing each shard task (BatchAi) assigned to it. These tasks are part of a larger task that is split into multiple small tasks for parallel processing.

[0099] Each shard task has its own specific task input parameters.

[0100] If a shard task is successfully executed, the executor returns the execution output of the task. This output usually contains processing result data or status information.

[0101] The sharding strategy for the next stream batch is determined based on the current stream batch, as follows:

[0102] For stream batches of the same group, if the next stream batch belongs to the same group, it is necessary to determine whether to perform secondary sharding on this new batch.

[0103] No secondary sharding is performed, that is, if it is decided not to subdivide, the subsequent stream batch will have the same number of shards (n) as the previous stream batch, and the task input parameters of each shard will be the execution output of the corresponding shard task of the previous stream batch.

[0104] If you perform secondary sharding, choosing to further subdivide, then for one shard of the previous stream batch (e.g., BatchAi), the subsequent stream batch may be divided into m small shards. In this case, the maximum number of shards of the subsequent stream batch will be the original number of shards n multiplied by the new number of shards m, that is, n*m. The input parameters of these m new shards will be the execution output of the original shard task.

[0105] If the next stream batch does not belong to the same group, the output of the shard task execution of the previous stream batch will not be used as its input parameter. This means that different groups can be executed independently of the previous task.

[0106] After completing a shard task, the executor will pass relevant information (including but not limited to execution output, status, etc.) back to the scheduler. This helps the scheduler understand the operating status of the entire system and make corresponding decisions or adjustments accordingly.

[0107] This mechanism allows the system to flexibly adjust task processing methods, appropriately sharding and reorganizing data based on actual needs, thereby optimizing resource utilization, improving processing speed, and ensuring data consistency and accuracy. Furthermore, this approach can also implement complex pipeline operations and support multi-stage data processing workflows.

[0108] In addition, once some or all of the shard tasks of a BatchAi (from BatchA) are completed, the scheduler will immediately save the shard task information corresponding to BatchB in the database based on the output of these shard tasks, and may generate one or more (m) new shard tasks (such as BatchBi1, BatchBi2, ..., BatchBim). This shows that the system can dynamically create subsequent tasks without waiting for all predecessor tasks to complete.

[0109] The newly generated shard tasks will be sent to the executor cluster for processing according to a specific allocation strategy (such as random allocation or polling, etc.). This approach improves the flexibility and resource utilization efficiency of the system.

[0110] When a shard task in a stream batch fails, other shard tasks that are being executed will continue to run, and the entire process will not be interrupted due to the failure of a single task.

[0111] The scanning thread monitors the status of all shard tasks in the stream batch. If all tasks except the failed task complete successfully, the overall status of the stream batch will be marked as failed.

[0112] If any shard task in the previous flow batch fails to execute, then even if all received tasks of the latter flow batch have been successfully executed, the latter will eventually be considered failed due to the lack of some input data passed from the previous flow batch.

[0113] To ensure accurate failure status propagation, the scanning thread will not officially update the status of the next flow batch to failure until it confirms that all available shard tasks from the previous flow batch have been processed.

[0114] This design ensures that even if some tasks fail, the system can continue to execute as much as possible while maintaining precise control over the overall job status, helping to improve the system's fault tolerance and stability. In addition, through real-time monitoring and adjustment, resources can be effectively managed, optimizing task execution order and load balancing.

[0115] First, the predecessor task is assigned to any server in the executor cluster for processing. This process is the starting point of the entire scheduling process, ensuring that each stream batch starts from a clear initial state. Upon successful execution, the server returns information containing several shard tasks and their input parameters. Based on this information, the system can further refine the task and generate a specific shard task list. Next, these shard tasks are distributed to different servers in the executor cluster according to a pre-defined task distribution strategy (such as random distribution or round-robin distribution). This step ensures balanced and efficient task processing across the cluster. After each executor completes its assigned shard task, the system collects all execution results and updates the status of the current stream batch based on these results. If all shard tasks complete successfully, the stream batch status is set to successful. If any shard task fails, the system waits until all possible shard tasks complete before changing the stream batch status to failed. Based on the execution results of the current stream batch, the system dynamically generates shard tasks for the next stream batch that depends on it, and repeats the above distribution and execution process. This mechanism allows the formation of complex dependency networks between tasks, namely DAG structures, thereby supporting more flexible and powerful workflow definitions.

[0116] In particular, when it comes to cross-batch task flow, input parameters can be dynamically adjusted based on the execution results of the previous batch of tasks. For example, within the same group, when the next batch is not resharded, its input parameters directly use the execution results of the previous batch; however, when resharding is required, all tasks after resharding will use the execution results of the previous batch as input.

[0117] Through the above steps, distributed DAG batch scheduling not only improves task processing efficiency and resource utilization, but also enhances support for complex business logic, making large-scale data processing and the implementation of complex business processes more efficient and flexible. This approach fully leverages the powerful computing power and distributed architecture of modern computing clusters, providing users with a powerful and flexible task scheduling solution.

[0118] Furthermore, the method of this embodiment passes tasks individually or in shards within a group of related stream batches, and upon completion, each task or shard can be immediately advanced to the next stream batch. This allows the specific parameters and business logic of subsequent operations to be dynamically determined based on the execution results of the previous stream batch, rather than pre-fixing all required information. Unlike traditional methods that require waiting for all shard tasks in the entire batch to complete before proceeding to the next step, this method allows the results of some shard tasks in the current stream batch to be passed to the next stream batch for processing as soon as they complete, thereby speeding up the overall process and improving resource utilization efficiency. If a shard task in a stream batch fails, the other shard tasks continue to run until all are completed. The system evaluates these results and sets the status of the stream batch to failed accordingly. If dependencies exist, even if all received tasks in the next stream batch complete successfully, it will ultimately be considered failed due to the lack of data from the previous stream batch. This mechanism ensures that status updates accurately reflect the actual execution status. Stream batches can be grouped and managed according to actual needs, ensuring that tasks within each group do not affect each other, enhancing the flexibility of business process design. This design is closer to real-world application scenarios and supports more complex workflow definitions.

[0119] With these features, this distributed DAG batch scheduling method not only improves the flexibility and responsiveness of batch job execution, but also adapts business operations to actual conditions, better meeting the needs of different application scenarios. This approach makes large-scale data processing and complex business logic implementation more efficient and easier to manage.

[0120] In order to transmit information between different batches during batch scheduling, reduce the waiting time during batch DAG execution, and improve resource utilization, the method of this embodiment follows the batch DAG call chain, supports distributed deployment, and transmits shard information (the number of shard tasks in the batch and the input / output parameters of each task) across batches. It also allows for customized secondary sharding processing, greatly improving the flexibility and overall efficiency of the system while also ensuring stability and consistency.

[0121] The above-mentioned distributed DAG batch scheduling method flexibly distributes the preceding tasks of the stream batch to any server in the executor cluster for processing, and dynamically creates specific shard tasks based on the returned shard task information after successful execution. This process enhances the system's adaptability and task refinement capabilities; according to the set task delivery strategy, the shard tasks are efficiently distributed to different executors in the cluster for processing. This method not only optimizes resource utilization, but also improves processing speed and load balancing; the execution results of all shard tasks are obtained and the status of the stream batch is updated accordingly, ensuring that even if a shard fails, the status of the entire stream batch can be accurately reflected, thereby maintaining the stability and consistency of the system; based on the results of the current stream batch, the shard tasks of the next stream batch are automatically generated, and the effective task delivery strategy is continued to be applied, realizing the automatic management of data flow and dependencies under complex business logic, and solving the problem of cross-batch information transmission; the overall method can not only handle complex shard tasks, but also flexibly respond to various scheduling requirements, significantly improving the responsiveness and processing efficiency in the face of complex task scenarios, and overcoming the shortcomings of existing technologies in this regard.

[0122] Figure 5 FIG is a schematic block diagram of a distributed DAG batch scheduling device 300 provided by an embodiment of the present invention. Figure 5 As shown, corresponding to the above distributed DAG batch scheduling method, the present invention also provides a distributed DAG batch scheduling device 300. The distributed DAG batch scheduling device 300 includes a unit for executing the above distributed DAG batch scheduling method, and the device can be configured in a server. Figure 5 The distributed DAG batch scheduling device 300 includes an allocating unit 301 , a shard receiving unit 302 , a creating unit 303 , a sending unit 304 , a result obtaining unit 305 , an updating unit 306 and a new batch task generating unit 307 .

[0123] The allocation unit 301 is used to obtain the preceding tasks of the stream batch and allocate them to any server in the executor cluster to obtain the execution result; the shard receiving unit 302 is used to receive the shard task information returned from the server when the execution result is successful execution; the creation unit 303 is used to create a number of shard tasks according to the shard task information; the dispatching unit 304 is used to dispatch a number of the shard tasks to the executor cluster for processing according to the set task dispatching strategy; the result acquisition unit 305 is used to obtain the execution results of the number of the shard tasks from the executor cluster; the update unit 306 is used to update the status of the stream batch according to the number of the execution results; the new batch task generation unit 307 is used to generate the shard tasks of the next stream batch that depends on the stream batch based on the number of the execution results, and execute the task dispatching strategy according to the set to dispatch a number of the shard tasks to the executor cluster for processing.

[0124] In one embodiment, the updating unit 306 is configured to set the current flow batch status to success when several execution results are all successful and when there is a previous flow batch and the status of the previous flow batch is success.

[0125] In one embodiment, the update unit 306 is used to set the execution status of the stream batch to failure when a shard task of the stream batch fails to execute, while other shard tasks remain in execution. After all tasks are completed, the execution status of the stream batch is set to failure. For the next stream batch that depends on the previous stream batch, the status of the next stream batch is changed to failure after ensuring that all possible shard tasks are completed.

[0126] It should be noted that those skilled in the art will clearly understand that the specific implementation process of the above-mentioned distributed DAG batch scheduling device 300 and each unit can refer to the corresponding description in the aforementioned method embodiment. For the convenience and brevity of the description, it will not be repeated here.

[0127] The distributed DAG batch scheduling device 300 can be implemented as a computer program. Figure 6 Runs on the computer equipment shown.

[0128] See also Figure 6 , Figure 6 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0129] See Figure 6The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0130] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which, when executed, can enable the processor 502 to perform a distributed DAG batch scheduling method.

[0131] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0132] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a distributed DAG batch scheduling method.

[0133] The network interface 505 is used to communicate with other devices through the network. Figure 6 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0134] The processor 502 is configured to execute a computer program 5032 stored in the memory to implement the following steps:

[0135] Obtain the preceding task of the stream batch and assign it to any server in the executor cluster to obtain the execution result; when the execution result is successful, receive the sharding task information returned from the server; create several sharding tasks according to the sharding task information; assign several sharding tasks to the executor cluster for processing according to the set task distribution strategy; obtain the execution results of several sharding tasks from the executor cluster; update the status of the stream batch according to several execution results; generate the sharding task of the next stream batch that depends on the stream batch based on several execution results, and execute the task distribution strategy according to the set task distribution strategy to assign several sharding tasks to the executor cluster for processing.

[0136] The shard task information includes a number of shard tasks into which the task corresponding to the stream batch is divided and input parameters of each shard task.

[0137] The task delivery strategy includes a random delivery strategy and a round-robin delivery strategy.

[0138] The rules for determining the input parameters of each shard task include:

[0139] When the next stream batch and the current stream batch are in the same group and the next stream batch is not split twice, the input parameter of each of the sharding tasks corresponding to the next stream batch is the execution result of the sharding task of the current stream batch.

[0140] When the next stream batch and the current stream batch are in the same group and the next stream batch is split twice, the input parameters of all sharding tasks of the next stream batch after the second split are the execution results of the sharding tasks of the current stream batch.

[0141] In one embodiment, when the processor 502 implements the step of updating the status of the flow batch according to the plurality of execution results, the processor 502 specifically implements the following steps:

[0142] When several execution results are all successful, and when there is a previous batch and the status of the previous batch is successful, the status of the current batch is set to successful.

[0143] In one embodiment, when the processor 502 implements the step of updating the status of the flow batch according to the plurality of execution results, the processor 502 specifically implements the following steps:

[0144] When a shard task in a stream batch fails to execute, other shard tasks remain in the execution state. After all tasks are completed, the stream batch execution state is set to failure. For the next stream batch that depends on the previous stream batch, the state of the next stream batch is changed to failure after ensuring that all possible shard tasks are completed.

[0145] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0146] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program includes program instructions, which can be stored in a storage medium that is computer-readable. The program instructions are executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0147] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor performs the following steps:

[0148] Obtain the preceding task of the stream batch and assign it to any server in the executor cluster to obtain the execution result; when the execution result is successful, receive the sharding task information returned from the server; create several sharding tasks according to the sharding task information; assign several sharding tasks to the executor cluster for processing according to the set task distribution strategy; obtain the execution results of several sharding tasks from the executor cluster; update the status of the stream batch according to several execution results; generate the sharding task of the next stream batch that depends on the stream batch based on several execution results, and execute the task distribution strategy according to the set task distribution strategy to assign several sharding tasks to the executor cluster for processing.

[0149] The shard task information includes a number of shard tasks into which the task corresponding to the stream batch is divided and input parameters of each shard task.

[0150] The task delivery strategy includes a random delivery strategy and a round-robin delivery strategy.

[0151] The rules for determining the input parameters of each shard task include:

[0152] When the next stream batch and the current stream batch are in the same group and the next stream batch is not split twice, the input parameter of each of the sharding tasks corresponding to the next stream batch is the execution result of the sharding task of the current stream batch.

[0153] When the next stream batch and the current stream batch are in the same group and the next stream batch is split twice, the input parameters of all sharding tasks of the next stream batch after the second split are the execution results of the sharding tasks of the current stream batch.

[0154] In one embodiment, when the processor executes the computer program to implement the step of updating the status of the flow batch according to the plurality of execution results, the processor specifically implements the following steps:

[0155] When several execution results are all successful, and when there is a previous batch and the status of the previous batch is successful, the status of the current batch is set to successful.

[0156] In one embodiment, when the processor executes the computer program to implement the step of updating the status of the flow batch according to the plurality of execution results, the processor specifically implements the following steps:

[0157] When a shard task in a stream batch fails to execute, other shard tasks remain in the execution state. After all tasks are completed, the stream batch execution state is set to failure. For the next stream batch that depends on the previous stream batch, the state of the next stream batch is changed to failure after ensuring that all possible shard tasks are completed.

[0158] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0159] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0160] In the several embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the various units is merely a logical functional division, and actual implementation may employ other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0161] The steps in the methods of the embodiments of the present invention may be adjusted in order, combined, or deleted as needed. The units in the devices of the embodiments of the present invention may be combined, divided, or deleted as needed. Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0162] If this integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present invention.

[0163] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. Distributed DAG batch scheduling method, characterized by: include: Obtain the preceding tasks of the stream batch and assign them to any server in the executor cluster to obtain the execution results; When the execution result is successful, receiving the sharding task information returned from the server; Create several sharding tasks according to the sharding task information; Allocate several of the shard tasks to the executor cluster for processing according to the set task delivery strategy; Obtaining execution results of the plurality of shard tasks from the executor cluster; updating the status of the flow batch according to a number of the execution results; Based on the plurality of execution results, a shard task of the next stream batch that depends on the stream batch is generated, and the set task delivery strategy is executed to distribute the plurality of shard tasks to the executor cluster for processing.

2. The distributed DAG batch scheduling method according to claim 1, characterized in that: The shard task information includes a number of shard tasks into which the task corresponding to the stream batch is divided and input parameters of each shard task.

3. The distributed DAG batch scheduling method according to claim 1, characterized in that: The task delivery strategy includes a random delivery strategy and a round-robin delivery strategy.

4. The distributed DAG batch scheduling method according to claim 2, characterized in that: The rules for determining the input parameters of each shard task include: When the next stream batch and the current stream batch are in the same group and the next stream batch is not split twice, the input parameter of each of the sharding tasks corresponding to the next stream batch is the execution result of the sharding task of the current stream batch.

5. The distributed DAG batch scheduling method according to claim 2, characterized in that: The rules for determining the input parameters of each shard task also include: When the next stream batch and the current stream batch are in the same group and the next stream batch is split twice, the input parameters of all sharding tasks of the next stream batch after the second split are the execution results of the sharding tasks of the current stream batch.

6. The distributed DAG batch scheduling method according to claim 1, characterized in that: The updating of the status of the flow batch according to the plurality of execution results comprises: When several execution results are all successful, and when there is a previous batch and the status of the previous batch is successful, the status of the current batch is set to successful.

7. The distributed DAG batch scheduling method according to claim 6, characterized in that: The updating of the state of the flow batch according to the plurality of execution results includes: When a shard task in a stream batch fails to execute, other shard tasks remain in the execution state. After all tasks are completed, the stream batch execution state is set to failure. For the next stream batch that depends on the previous stream batch, the state of the next stream batch is changed to failure after ensuring that all possible shard tasks are completed.

8. Distributed DAG batch scheduling device, characterized by: include: The allocation unit is used to obtain the preceding tasks of the stream batch and allocate them to any server in the executor cluster to obtain the execution results; A shard receiving unit, configured to receive shard task information returned from the server when the execution result is successful; A creating unit, configured to create a plurality of sharding tasks according to the sharding task information; A dispatching unit, configured to dispatch a plurality of the shard tasks to an executor cluster for processing according to a set task dispatching strategy; A result acquisition unit, configured to acquire execution results of the plurality of shard tasks from the executor cluster; an updating unit, configured to update the state of the flow batch according to a plurality of the execution results; A new batch task generation unit is used to generate shard tasks of the next stream batch that depends on the stream batch based on the plurality of execution results, and to execute the set task delivery strategy to distribute the plurality of shard tasks to the executor cluster for processing.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images