Task scheduling method and device for graph processing request
By building a state machine migration diagram and a unified scheduling diagram to process requests, the problem of lack of unified management of task scheduling in the existing technology is solved, and efficient scheduling and management of task units in multiple modes is achieved.
Patent Information
- Application Number
- CN202510644860.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-19
- Publication Date
- 2025-08-08
AI Technical Summary
In the prior art, task scheduling for graph processing requests cannot effectively manage the dependencies between multiple computing modes, resulting in independent scheduling and execution of subtasks and lack of unified scheduling management.
By building a state machine migration graph, the task units in multiple computing modes are uniformly dispatched, and the nodes and migration edges in the state machine migration graph are used to manage subtasks in multiple modes of task unit scheduling.
It realizes unified scheduling and management of task units in multiple computing modes, solves the scenario where the execution process mode is variable and requires the combination of multiple consecutive subtasks, and improves the efficiency and flexibility of task scheduling.
Smart Images
Figure CN120448130A_ABST
Abstract
Description
Technical Field
[0001] One or more embodiments of this specification relate to the field of computers, and more particularly, to a method and apparatus for scheduling tasks for graph processing requests. Background Art
[0002] Graph computing engines require computing capabilities in multiple modes, including stream processing for stream computing, batch processing for batch computing, and graph iteration for graph computing. Using scheduling basic units as a unified scheduling description structure, these multiple modes can be uniformly scheduled and executed. Each scheduling basic unit is a multi-round cyclic scheduling model. The execution of a scheduling basic unit consists of a series of task units to be executed, which can be split into a series of subtasks.
[0003] In the existing technology, for task scheduling of graph processing requests, after splitting a task unit of a pattern into a series of continuous subtasks, each subtask can only be scheduled for execution independently. There is no clear connection between the subtasks, and the execution of each subtask is dependent on each other. Summary of the Invention
[0004] One or more embodiments of this specification describe a method and apparatus for scheduling tasks for graph processing requests, which can perform unified scheduling management on a series of subtasks split into multiple modes.
[0005] In a first aspect, a task scheduling method for a graph processing request is provided, the method comprising:
[0006] Acquire multiple task units corresponding to the target graph processing request; the multiple task units correspond to multiple computing modes, including a graph iteration computing mode, and at least one of a stream processing mode and a batch processing mode;
[0007] Obtaining a state machine transition diagram uniformly constructed for the multiple computing modes; the state machine transition diagram includes multiple nodes, each node indicates the execution status of a subtask split from each computing mode, and the transition edges between the nodes correspond to the subtasks, and the subtasks included in any computing mode have corresponding transition edges in the state machine transition diagram;
[0008] According to the state machine transition diagram, the multiple task units are scheduled for execution, wherein the scheduling and execution of any target task unit includes: in the first state indicated by the first node, in response to satisfying the first trigger condition for starting migration to the second state, scheduling the execution of the target job in the target task unit, thereby migrating the current state to the second state; the target job belongs to the target subtask corresponding to the migration edge between the first node and the second node corresponding to the second state.
[0009] In one possible embodiment, the multiple task units include at least a first task unit and a second task unit, the first task unit corresponds to a graph iteration computing mode, the second task unit corresponds to data processing before or after graph iteration, and the data processing adopts a streaming processing mode or a batch processing mode.
[0010] In a possible implementation, any two different computing modes have at least one common head-tail node triplet in the state machine transition diagram.
[0011] In a possible implementation, the batch processing mode is divided into an initialization subtask, a calculation subtask, and a result processing subtask;
[0012] The streaming processing mode is divided into an initialization subtask and a calculation subtask that is executed multiple times;
[0013] The graph iterative computing mode is divided into an initialization subtask, a graph loading subtask, a multi-execution computing subtask, and a result processing subtask.
[0014] In a possible implementation, the multiple nodes respectively represent a start state, an initialization state, a graph loaded state, a processing state, a completed state, and an end state, wherein the processing state is a node state that can be repeatedly executed.
[0015] Furthermore, the target task unit is in graph iterative computing mode, streaming processing mode or batch processing mode; the first state is a start state, the target subtask is an initialization subtask, and the second state is an initialization state.
[0016] Furthermore, the target task unit is in a graph iterative calculation mode; the first state is an initialization state, the target subtask is a graph loading subtask, and the second state is a graph loaded state.
[0017] Furthermore, the target task unit is in batch processing mode or stream processing mode; the first state is an initialization state, the target subtask is a calculation subtask, and the second state is a processing state.
[0018] Furthermore, the target task unit is in a graph iterative calculation mode; the first state is a graph loaded state, the target subtask is a calculation subtask, and the second state is a processing state.
[0019] Furthermore, the target task unit is in streaming processing mode or graph iterative computing mode; the first state is the processing state, the first trigger condition is the completion of the current round of calculation, and the current round is not the last round, the target subtask is the computing subtask, and the second state is the processing state.
[0020] Furthermore, the target task unit is a graph iterative calculation mode; the first state is a processing state, the first trigger condition is the completion of all rounds of calculation, the target subtask is a result processing subtask, and the second state is an end state.
[0021] In a possible implementation, the scheduling execution of the target job in the target task unit includes:
[0022] Create a set of distributed scheduling subtasks according to the first state, the second state, and the information of the target job;
[0023] Send a set of distributed scheduling subtasks to the worker nodes for execution.
[0024] Furthermore, the step of migrating the current state to the second state includes:
[0025] Get the execution results of a set of distributed scheduling subtasks;
[0026] The execution of a group of distributed scheduling subtasks is determined to be completed according to the execution result, so as to determine that the execution of the target job is completed.
[0027] In a second aspect, a task scheduling device for graph processing requests is provided, the device comprising:
[0028] A first acquisition unit is configured to acquire a plurality of task units corresponding to a target graph processing request; the plurality of task units correspond to a plurality of computing modes, including a graph iteration computing mode, and at least one of a stream processing mode and a batch processing mode;
[0029] A second acquisition unit is configured to acquire a state machine transition diagram uniformly constructed for the multiple computing modes; the state machine transition diagram includes a plurality of nodes, each node indicating an execution state of a subtask split from each computing mode, and transition edges between the nodes corresponding to the subtasks, and the subtasks included in any computing mode have corresponding transition edges in the state machine transition diagram;
[0030] A scheduling unit is used to schedule the execution of multiple task units obtained by the first acquisition unit according to the state machine transition diagram obtained by the second acquisition unit, wherein the scheduling execution of any target task unit includes: in the first state indicated by the first node, in response to satisfying the first trigger condition for starting migration to the second state, scheduling the execution of the target job in the target task unit, thereby migrating the current state to the second state; the target job belongs to the target subtask corresponding to the migration edge between the first node and the second node corresponding to the second state.
[0031] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method according to the first aspect.
[0032] In a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.
[0033] Through the method and device provided in the embodiments of this specification, first, multiple task units corresponding to the target graph processing request are obtained; the multiple task units correspond to multiple computing modes, including a graph iteration computing mode, and at least one of a streaming processing mode and a batch processing mode; then a state machine transition diagram uniformly constructed for the multiple computing modes is obtained; the state machine transition diagram includes multiple nodes, each node indicates the execution status of a subtask split from each computing mode, and the migration edges between the nodes correspond to the subtasks, and the subtasks contained in any computing mode have corresponding migration edges in the state machine transition diagram; finally, according to the state machine transition diagram, the multiple task units are scheduled for execution, wherein the scheduling execution of any target task unit includes: in the first state indicated by the first node, in response to satisfying the first trigger condition for starting migration to the second state, scheduling the execution of the target job in the target task unit, thereby migrating the current state to the second state; the target job belongs to the target subtask corresponding to the migration edge between the first node and the second node corresponding to the second state. As can be seen from the above, the embodiments of this specification integrate the scheduling processes of multiple modes into a state machine, complete the distribution and execution of a series of subtasks through state change migration in the state machine, complete the task scheduling of complex processes, and can solve the scenarios where the execution process modes are changeable and the execution of tasks requires a combination of multiple subtasks in a row. It can also perform unified scheduling management for a series of subtasks split into multiple modes. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0035] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;
[0036] Figure 2 A flowchart of a method for scheduling tasks for graph processing requests according to one embodiment is shown;
[0037] Figure 3 shows a state machine transition diagram according to one embodiment;
[0038] Figure 4 A schematic diagram of a scheduling process based on state migration according to an embodiment is shown;
[0039] Figure 5 A schematic block diagram of a task scheduling apparatus for graph processing requests according to one embodiment is shown. DETAILED DESCRIPTION
[0040] The solution provided in this specification is described below in conjunction with the accompanying drawings.
[0041] Figure 1 This is a schematic diagram of an implementation scenario of an embodiment disclosed in this specification. This implementation scenario involves task scheduling for a graph processing request. The graph processing request can be converted into multiple task units, so that the execution of multiple task units can be scheduled. The multiple task units correspond to multiple computing modes, including a graph iteration computing mode, and at least one of a stream processing mode and a batch processing mode. Figure 1 ,Graph iterative computing mode, streaming processing mode and batch processing mode can be divided into several subtasks respectively.
[0042] Among them, the batch processing mode is used for batch computing, directly sending a BatchTask subtask, which completely contains the required initialization information (Init), computing information (Compute), and result processing (Finish) after the calculation is completed.
[0043] Streaming processing mode is used for streaming computing. It is necessary to send the InitTask subtask to initialize all subtasks, and then send the ComputeTask subtask to perform calculations repeatedly. Each calculation processes a fixed batch of data. A batch of data can be fixed in size or fixed in time.
[0044] The graph iteration computing mode is used for graph computing. It is necessary to send the InitTask subtask to initialize all subtasks, then execute the LoadGraph subtask to load the vertex edges into the existing graph, and then repeatedly execute the ComputeTask subtask according to the graph iteration situation. Finally, it is necessary to execute the FinishTask subtask to send the results of the graph computing to the downstream task unit or return it to the user.
[0045] In any of the aforementioned computing modes, the execution of each subtask is dependent on the execution of the previous subtask. For example, the ComputeTask subtask requires the InitTask subtask to complete first, and the FinishTask subtask also needs to wait for the previous subtasks to complete before receiving the result. Therefore, it can be thought of as splitting a task into a series of sequentially executed subtasks. The embodiments of this specification utilize a scheduling state machine to implement unified scheduling and management of this series of subtasks.
[0046] Among them, the state machine refers to a mathematical model of state transition, which fully describes the transition diagram from the initial state to the final state after a series of state transitions.
[0047] Figure 2 A flowchart of a task scheduling method for graph processing requests according to an embodiment is shown. The method can be based on Figure 1 The implementation scenario shown in Figure 2 As shown, the task scheduling method for graph processing requests in this embodiment includes the following steps: Step 21, obtaining multiple task units corresponding to the target graph processing request; the multiple task units correspond to multiple computing modes, including graph iteration computing mode, and at least one of stream processing mode and batch processing mode; Step 22, obtaining a state machine transition diagram uniformly constructed for the multiple computing modes; the state machine transition diagram includes multiple nodes, each node indicates the execution state of the subtask split from each computing mode, the migration edge between the nodes corresponds to the subtask, and the subtasks contained in any computing mode have corresponding migration edges in the state machine transition diagram; Step 23, scheduling the execution of the multiple task units according to the state machine transition diagram, wherein the scheduling execution of any target task unit includes: in the first state indicated by the first node, in response to satisfying the first trigger condition for starting migration to the second state, scheduling the execution of the target job in the target task unit, thereby migrating the current state to the second state; the target job belongs to the target subtask corresponding to the migration edge between the first node and the second node corresponding to the second state. The specific execution method of each of the above steps is described below.
[0048] First, in step 21, multiple task units corresponding to the target graph processing request are obtained; these multiple task units correspond to multiple computing modes, including a graph iteration computing mode, and at least one of a streaming processing mode and a batch processing mode. It is understood that during the computation and processing of graph data, different processing modes may be employed, including streaming graph composition, batch graph composition, static graph computation, dynamic graph computation, and so on.
[0049] In one example, the multiple task units include at least a first task unit and a second task unit, the first task unit corresponds to a graph iteration computing mode, and the second task unit corresponds to data processing before or after graph iteration, and the data processing adopts a streaming processing mode or a batch processing mode.
[0050] In this example, corresponding to a typical scenario of graph data analysis and processing, it is necessary to first construct a graph, then perform graph calculations on the constructed graph data structure, and then perform necessary post-processing on the graph calculation results.
[0051] Graph construction involves extracting the vertex and edge structure of graph data from existing data, such as relational data or unstructured data, and constructing it into a graph data structure. Graph construction can be performed in a streaming or batch manner. For example, streaming construction involves continuously reading small batches of data from a data source (e.g., a large data table) to form some vertex / edge structures. Batch construction involves, for example, reading specified target data once and forming a graph data structure based on the target data.
[0052] Based on the constructed graph data structure, graph computations can be performed, such as extracting subgraphs and calculating connectivity between nodes. Graph computations can be performed once on an existing static graph, or as the graph is constructed, enabling dynamic graph computations. Post-processing involves further transformation and processing of the graph computation results before ultimately writing them to the result table.
[0053] Then, in step 22, a state machine transition diagram uniformly constructed for the multiple computing modes is obtained; the state machine transition diagram includes multiple nodes, each node indicating the execution status of a subtask split from each computing mode, and the transition edges between the nodes correspond to the subtasks. The subtasks included in any computing mode have corresponding transition edges in the state machine transition diagram. It is understood that the scheduling process of any computing mode can be carried out based on this state machine transition diagram.
[0054] In one example, any two different computing modes have at least one common head-tail node triplet in the state machine transition diagram.
[0055] In this example, when constructing the state machine transition diagram, existing states and executed subtasks are reused as much as possible across different computing modes to increase code reuse. Furthermore, through fine-grained subtask splitting and combination, subtasks that implement complex functions can be combined by combining simple subtasks, similar to building blocks.
[0056] In one example, the batch processing mode is divided into an initialization subtask, a calculation subtask, and a result processing subtask;
[0057] The streaming processing mode is divided into an initialization subtask and a calculation subtask that is executed multiple times;
[0058] The graph iterative computing mode is divided into an initialization subtask, a graph loading subtask, a multi-execution computing subtask, and a result processing subtask.
[0059] In this example, for the batch processing mode, a typical independent BatchTask subtask is split into fine-grained subtasks according to actual functions. This unifies the entire execution model. Regardless of the streaming processing mode, batch processing mode, or graph iterative computing mode, existing states and executed subtasks can be reused as much as possible, increasing code reuse capabilities.
[0060] In one example, the multiple nodes respectively represent a start state, an initialization state, a graph loaded state, a processing state, a completed state, and an end state, wherein the processing state is a node state that can be executed repeatedly.
[0061] In this example, the Initialization state indicates that the Initialization subtask has been completed, the Graph Loaded state indicates that the Graph Loading subtask has been completed, the Processing state indicates that a Calculation subtask has been completed, and the Completed state indicates that the Result Processing subtask has been completed.
[0062] Figure 3 FIG. 1 shows a state machine transition diagram according to an embodiment. Figure 3 Based on the scheduling processes corresponding to the streaming, batch, and graph iterative computing modes, a unified state machine transition diagram is constructed to implement a series of complex task scheduling and management. START represents the start state, Inited represents the initialization state, GraphLoaded represents the graph loaded state, Processing represents the processing state, Finished represents the completed state, and END represents the end state. InitTASK represents the initialization subtask, LoadGraph represents the graph loading subtask, ComputeTASK represents the computation subtask, and FinishTASK represents the result processing subtask.
[0063] In the state machine transition diagram of the unified state machine, the execution processes of the streaming processing mode, batch processing mode, and graph iterative computing mode are:
[0064] Batch processing mode, START->InitTask->ComputeTask->FinishTask->END;
[0065] Streaming mode, START->InitTask->ComputeTask (multiple executions)->END;
[0066] Graph iterative computing mode: START->InitTask->LoadGraphTask->ComputeTask (multiple executions)->FinishTask->END.
[0067] It should be noted that in this state machine transition diagram, the execution process of the batch processing mode includes InitTask->ComputeTask->FinishTask. Compared with the original subtask being sent only once, the embodiment of this specification splits the original independent BatchTask subtask into fine-grained parts according to the actual function.
[0068] Finally, in step 23, the multiple task units are scheduled for execution based on the state machine transition diagram. Scheduling any target task unit includes: in a first state indicated by a first node, in response to satisfying a first trigger condition for starting a transition to a second state, scheduling the execution of a target job in the target task unit, thereby migrating the current state to the second state; the target job belongs to the target subtask corresponding to the transition edge between the first node and the second node corresponding to the second state. It will be appreciated that the target job may include one or more logical operators, such as a union operator, a filter operator, a join operator, and the like.
[0069] In the embodiments of this specification, task scheduling and execution can be implemented between two states according to the state machine transition diagram. A state transition in the state machine includes the following elements:
[0070] Current state: the current state;
[0071] Target state: The target state of the migration after the trigger condition is met;
[0072] Migration trigger event (event): The trigger condition that needs to be met to migrate from the current state to the target state;
[0073] Action: The action that needs to be performed to move from the current state to the target state.
[0074] A state transition in a state machine can be described as: when a specified trigger event is met, a series of actions can be performed to migrate the current state to the target state.
[0075] In one example, the target task unit is in graph iteration computing mode, streaming processing mode, or batch processing mode; the first state is a start state, the target subtask is an initialization subtask, and the second state is an initialization state.
[0076] In this example, if the current state is the start state, no matter which computing mode the target task unit is in, the target state is the initialization state, and the execution action is to execute the initialization subtask.
[0077] In one example, the target task unit is in a graph iteration calculation mode; the first state is an initialization state, the target subtask is a graph loading subtask, and the second state is a graph loaded state.
[0078] In this example, if the current state is the initialization state and the target task unit is the graph iteration calculation mode, the target state is the graph loaded state, and the execution action is to execute the graph loading subtask.
[0079] In one example, the target task unit is in batch processing mode or stream processing mode; the first state is an initialization state, the target subtask is a computing subtask, and the second state is a processing state.
[0080] In this example, if the current state is the initialization state and the target task unit is in batch processing mode or stream processing mode, the target state is the processing state, and the execution action is to execute the calculation subtask.
[0081] In one example, the target task unit is in a graph iterative computing mode; the first state is a graph loaded state, the target subtask is a computing subtask, and the second state is a processing state.
[0082] In this example, if the current state is the graph loaded state and the target task unit is the graph iterative calculation mode, the target state is the processing state and the execution action is to execute the calculation subtask.
[0083] In one example, the target task unit is in streaming processing mode or graph iterative computing mode; the first state is the processing state, the first trigger condition is the completion of the current round of calculation, and the current round is not the last round, the target subtask is the computing subtask, and the second state is the processing state.
[0084] In this example, if the current state is the processing state, and the target task unit is the streaming processing mode or the graph iteration calculation mode, the first trigger condition is to complete the current round of calculation, and the current round is not the last round, then the target state is the processing state, and the execution action is to execute the calculation subtask.
[0085] In one example, the target task unit is a graph iterative calculation mode; the first state is a processing state, the first trigger condition is the completion of all rounds of calculation, the target subtask is a result processing subtask, and the second state is an end state.
[0086] In this example, if the current state is the processing state and the target task unit is the graph iteration calculation mode, the first trigger condition is the completion of all rounds of calculation, then the target state is the end state, and the execution action is to execute the result processing subtask.
[0087] In one example, the scheduling execution of the target job in the target task unit includes:
[0088] Create a set of distributed scheduling subtasks according to the first state, the second state, and the information of the target job;
[0089] Send a set of distributed scheduling subtasks to the worker nodes for execution.
[0090] In this example, distributed execution of the target job can be achieved, improving execution efficiency.
[0091] Furthermore, the step of migrating the current state to the second state includes:
[0092] Get the execution results of a set of distributed scheduling subtasks;
[0093] The execution of a group of distributed scheduling subtasks is determined to be completed according to the execution result, so as to determine that the execution of the target job is completed.
[0094] In this example, after it is determined that the target job is completed, the current state may be migrated to the second state.
[0095] Figure 4 FIG. 1 shows a schematic diagram of a scheduling process based on state migration according to an embodiment. Figure 4 , the current state is the first state, which can be recorded as State1, the target state is the second state, which can be recorded as State2, the triggering event after the migration condition is met is event1, and the action to be performed for state migration is action1. The scheduling components involved include the state machine manager (StateMachineManager), the task scheduler (Scheduler), and the worker node (worker). The state machine manager is used to manage all migration states and migration conditions of the state machine. When the migration conditions are met, a state migration will be triggered. The task scheduler is used to schedule the execution of the current job, split the job into a series of subtasks (tasks) that can be executed in a distributed manner, and send them to the worker node for execution. At the same time, the execution results of all subtasks will be collected. The worker node is a container for distributed execution, deployed on different machines, which can receive subtasks sent by the task scheduler, execute them, and return the results.
[0096] A task scheduling process based on state transitions can include: first, the state machine manager checks the current state machine and finds that the conditions for transitioning from the first state to the second state are met, such as event1, triggering a state transition. Then, action1 is executed, which calls the task scheduler, notifying the task scheduler to perform a round of scheduling. The task scheduler then creates a set of distributed subtasks based on the current state, target state, and action1 information, and sends them to remote worker nodes for execution. The remote worker nodes receive the results of each subtask and return them to the task scheduler. After receiving the message that all subtasks have completed execution, the task scheduler notifies the state machine manager, triggers a check migration, and continues with the next scheduling. This fully implements the scheduling process triggered by state transitions.
[0097] In addition, it should be noted that after the task scheduler creates a set of distributed subtasks and sends them to the remote worker node for execution, it can decide whether to wait for the completion of all subtasks based on the current information before notifying the state machine manager to trigger the migration check. In other words, the task scheduler can notify the state machine manager to perform a status check without waiting for the completion of all subtasks. If the conditions for triggering the next round of scheduling are currently met, the next round of scheduling will be triggered immediately. This mechanism realizes the asynchronous execution of task scheduling, and there is no need to synchronously wait for the task corresponding to the previous action to be completed before triggering the next state migration action. At the same time, there will be no performance regression after splitting the batch processing mode from a subtask into two more fine-grained consecutive subtasks.
[0098] Through the method provided in the embodiments of this specification, first, multiple task units corresponding to the target graph processing request are obtained; the multiple task units correspond to multiple computing modes, including a graph iteration computing mode, and at least one of a streaming processing mode and a batch processing mode; then a state machine transition diagram uniformly constructed for the multiple computing modes is obtained; the state machine transition diagram includes multiple nodes, each node indicates the execution status of a subtask split from each computing mode, and the migration edges between the nodes correspond to the subtasks, and the subtasks contained in any computing mode have corresponding migration edges in the state machine transition diagram; finally, according to the state machine transition diagram, the multiple task units are scheduled for execution, wherein the scheduling execution of any target task unit includes: in the first state indicated by the first node, in response to satisfying the first trigger condition for starting migration to the second state, scheduling the execution of the target job in the target task unit, thereby migrating the current state to the second state; the target job belongs to the target subtask corresponding to the migration edge between the first node and the second node corresponding to the second state. As can be seen from the above, the embodiments of this specification integrate the scheduling processes of multiple modes into a state machine, complete the distribution and execution of a series of subtasks through state change migration in the state machine, complete the task scheduling of complex processes, and can solve the scenarios where the execution process modes are changeable and the execution of tasks requires a combination of multiple subtasks in a row. It can also perform unified scheduling management for a series of subtasks split into multiple modes.
[0099] According to another aspect of the embodiment, a task scheduling device for graph processing requests is also provided, and the device is used to execute the method provided in the embodiment of this specification. Figure 5 FIG. 1 is a schematic block diagram of a task scheduling apparatus for graph processing requests according to an embodiment. Figure 5 As shown, the apparatus 500 includes:
[0100] A first acquiring unit 51 is configured to acquire a plurality of task units corresponding to a target graph processing request; the plurality of task units corresponding to a plurality of computing modes, including a graph iteration computing mode, and at least one of a stream processing mode and a batch processing mode;
[0101] A second acquisition unit 52 is configured to acquire a state machine transition diagram uniformly constructed for the multiple computing modes; the state machine transition diagram includes multiple nodes, each node indicating the execution status of a subtask split from each computing mode, and the transition edges between the nodes correspond to the subtasks. The subtasks included in any computing mode have corresponding transition edges in the state machine transition diagram;
[0102] The scheduling unit 53 is used to schedule the execution of multiple task units obtained by the first acquisition unit 51 according to the state machine transition diagram obtained by the second acquisition unit 52, wherein the scheduling execution of any target task unit includes: in the first state indicated by the first node, in response to satisfying the first trigger condition for starting migration to the second state, scheduling the execution of the target job in the target task unit, thereby migrating the current state to the second state; the target job belongs to the target subtask corresponding to the migration edge between the first node and the second node corresponding to the second state.
[0103] Optionally, as an embodiment, the multiple task units include at least a first task unit and a second task unit, the first task unit corresponds to a graph iteration calculation mode, and the second task unit corresponds to data processing before or after graph iteration, and the data processing adopts a streaming processing mode or a batch processing mode.
[0104] Optionally, as an embodiment, any two different computing modes have at least one common head and tail node triplet in the state machine transition diagram.
[0105] Optionally, as an embodiment, the batch processing mode is divided into an initialization subtask, a calculation subtask, and a result processing subtask;
[0106] The streaming processing mode is divided into an initialization subtask and a calculation subtask that is executed multiple times;
[0107] The graph iterative computing mode is divided into an initialization subtask, a graph loading subtask, a multi-execution computing subtask, and a result processing subtask.
[0108] Optionally, as an embodiment, the multiple nodes respectively represent a start state, an initialization state, a graph loaded state, a processing state, a completed state and an end state, wherein the processing state is a node state that can be executed repeatedly.
[0109] Furthermore, the target task unit is in graph iterative computing mode, streaming processing mode or batch processing mode; the first state is a start state, the target subtask is an initialization subtask, and the second state is an initialization state.
[0110] Furthermore, the target task unit is in a graph iterative calculation mode; the first state is an initialization state, the target subtask is a graph loading subtask, and the second state is a graph loaded state.
[0111] Furthermore, the target task unit is in batch processing mode or stream processing mode; the first state is an initialization state, the target subtask is a calculation subtask, and the second state is a processing state.
[0112] Furthermore, the target task unit is in a graph iterative calculation mode; the first state is a graph loaded state, the target subtask is a calculation subtask, and the second state is a processing state.
[0113] Furthermore, the target task unit is in streaming processing mode or graph iterative computing mode; the first state is the processing state, the first trigger condition is the completion of the current round of calculation, and the current round is not the last round, the target subtask is the computing subtask, and the second state is the processing state.
[0114] Furthermore, the target task unit is a graph iterative calculation mode; the first state is a processing state, the first trigger condition is the completion of all rounds of calculation, the target subtask is a result processing subtask, and the second state is an end state.
[0115] Optionally, as an embodiment, the scheduling unit 53 includes:
[0116] A creation subunit is used to create a set of distributed scheduling subtasks according to the first state, the second state, and information of the target job;
[0117] The sending sub-unit is used to send a group of distributed scheduling sub-tasks created by the creating sub-unit to the working node for execution.
[0118] Furthermore, the scheduling unit 53 includes:
[0119] The acquisition sub-unit is used to obtain the execution results of a group of distributed scheduling subtasks;
[0120] The determining subunit is used to determine the completion of execution of a group of distributed scheduling subtasks according to the execution results obtained by the obtaining subunit, so as to determine the completion of execution of the target job.
[0121] Through the device provided by the embodiments of this specification, first, the first acquisition unit 51 acquires multiple task units corresponding to the target graph processing request; the multiple task units correspond to multiple computing modes, including a graph iteration computing mode, and at least one of a streaming processing mode and a batch processing mode; then the second acquisition unit 52 acquires a state machine transition diagram uniformly constructed for the multiple computing modes; the state machine transition diagram includes multiple nodes, each node indicates the execution status of a subtask split from each computing mode, and the migration edges between the nodes correspond to the subtasks, and the subtasks contained in any computing mode have corresponding migration edges in the state machine transition diagram; finally, the scheduling unit 53 schedules the execution of the multiple task units according to the state machine transition diagram, wherein the scheduling execution of any target task unit includes: in the first state indicated by the first node, in response to satisfying the first trigger condition for starting migration to the second state, scheduling the execution of the target job in the target task unit, thereby migrating the current state to the second state; the target job belongs to the target subtask corresponding to the migration edge between the first node and the second node corresponding to the second state. As can be seen from the above, the embodiments of this specification integrate the scheduling processes of multiple modes into a state machine, complete the distribution and execution of a series of subtasks through state change migration in the state machine, complete the task scheduling of complex processes, and can solve the scenarios where the execution process modes are changeable and the execution of tasks requires a combination of multiple subtasks in a row. It can also perform unified scheduling management for a series of subtasks split into multiple modes.
[0122] According to another embodiment, there is also provided a computer readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 2 The method described.
[0123] According to another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the system realizes the combination of Figure 2 The method described.
[0124] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.
[0125] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.
Claims
1. A task scheduling method for a graph processing request, the method comprising: Get multiple task units corresponding to the target graph processing request; The plurality of task units correspond to a plurality of computing modes, including a graph iteration computing mode, and at least one of a stream processing mode and a batch processing mode; Obtaining a state machine transition diagram uniformly constructed for the multiple computing modes; the state machine transition diagram includes multiple nodes, each node indicates the execution status of a subtask split from each computing mode, and the transition edges between the nodes correspond to the subtasks, and the subtasks included in any computing mode have corresponding transition edges in the state machine transition diagram; According to the state machine transition diagram, the multiple task units are scheduled for execution, wherein the scheduling and execution of any target task unit includes: in the first state indicated by the first node, in response to satisfying the first trigger condition for starting migration to the second state, scheduling the execution of the target job in the target task unit, thereby migrating the current state to the second state; the target job belongs to the target subtask corresponding to the migration edge between the first node and the second node corresponding to the second state.
2. The method according to claim 1, wherein The multiple task units include at least a first task unit and a second task unit, the first task unit corresponds to a graph iteration calculation mode, and the second task unit corresponds to data processing before or after graph iteration, and the data processing adopts a streaming processing mode or a batch processing mode.
3. The method according to claim 1, wherein Any two different computing modes have at least one common head-tail node triplet in the state machine transition diagram.
4. The method according to claim 1, wherein The batch processing mode is divided into an initialization subtask, a calculation subtask, and a result processing subtask; The streaming processing mode is divided into an initialization subtask and a calculation subtask that is executed multiple times; The graph iterative computing mode is divided into an initialization subtask, a graph loading subtask, a multi-execution computing subtask, and a result processing subtask.
5. The method according to claim 1, wherein The multiple nodes respectively represent a start state, an initialization state, a graph loaded state, a processing state, a completed state and an end state, wherein the processing state is a node state that can be repeatedly executed.
6. The method according to claim 5, wherein: The target task unit is in graph iteration computing mode, streaming processing mode or batch processing mode; the first state is the start state, the target subtask is the initialization subtask, and the second state is the initialization state.
7. The method according to claim 5, wherein: The target task unit is in a graph iterative calculation mode; the first state is an initialization state, the target subtask is a graph loading subtask, and the second state is a graph loaded state.
8. The method of claim 5, wherein: The target task unit is in batch processing mode or stream processing mode; the first state is an initialization state, the target subtask is a calculation subtask, and the second state is a processing state.
9. The method of claim 5, wherein: The target task unit is in graph iterative calculation mode; the first state is the graph loaded state, the target subtask is the calculation subtask, and the second state is the processing state.
10. The method of claim 5, wherein: The target task unit is in streaming processing mode or graph iterative computing mode; the first state is the processing state, the first trigger condition is the completion of the current round of calculation, and the current round is not the last round, the target subtask is the computing subtask, and the second state is the processing state.
11. The method according to claim 5, wherein: The target task unit is a graph iterative calculation mode; the first state is a processing state, the first trigger condition is the completion of all rounds of calculation, the target subtask is a result processing subtask, and the second state is an end state.
12. The method of claim 1, wherein: The scheduling and executing of the target job in the target task unit includes: Create a set of distributed scheduling subtasks according to the first state, the second state, and the information of the target job; Send a set of distributed scheduling subtasks to the worker nodes for execution.
13. The method of claim 12, wherein: The step of migrating the current state to the second state includes: Get the execution results of a set of distributed scheduling subtasks; The execution of a group of distributed scheduling subtasks is determined to be completed according to the execution result, so as to determine that the execution of the target job is completed.
14. A task scheduling device for graph processing requests, the device comprising: A first acquiring unit, configured to acquire a plurality of task units corresponding to a target graph processing request; The plurality of task units correspond to a plurality of computing modes, including a graph iteration computing mode, and at least one of a stream processing mode and a batch processing mode; A second acquisition unit is configured to acquire a state machine transition diagram uniformly constructed for the multiple computing modes; the state machine transition diagram includes a plurality of nodes, each node indicating an execution state of a subtask split from each computing mode, and transition edges between the nodes corresponding to the subtasks, and the subtasks included in any computing mode have corresponding transition edges in the state machine transition diagram; A scheduling unit is used to schedule the execution of multiple task units obtained by the first acquisition unit according to the state machine transition diagram obtained by the second acquisition unit, wherein the scheduling execution of any target task unit includes: in the first state indicated by the first node, in response to satisfying the first trigger condition for starting migration to the second state, scheduling the execution of the target job in the target task unit, thereby migrating the current state to the second state; the target job belongs to the target subtask corresponding to the migration edge between the first node and the second node corresponding to the second state.
15. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 13.
16. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 13 is implemented.