Scheduling method and apparatus

Abstract

The embodiment of the application provides a scheduling method, which is used for a DAG scheduling server, and the method comprises the following steps: receiving a calling request, wherein the calling request comprises a request parameter used for representing a calling trigger mode; obtaining a target DAG according to the scheduling request, wherein the target DAG comprises a plurality of processing units with a dependency relationship; and performing target scheduling according to the calling trigger mode and the target DAG, so as to execute a processing unit to be executed in the plurality of units by an executor. The scheduling method provided by the embodiment of the application performs different scheduling according to the calling trigger mode and the target DAG, improves the flexibility of the scheduling service, and improves the operation of the video processing platform and the video processing efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a scheduling method, apparatus, computer device, and computer-readable storage medium. Background Technology

[0002] With the development of video platforms, video processing primarily uses DATABUS messages and HTTP / RPC interfaces as the main communication methods. Business logic is distributed across different servers. Video processing includes video transcoding and review, involving numerous messages and complex processes. The inventors understand that video processing typically involves scheduling a large number of processing functions (processing logic), but existing scheduling services lack flexibility, thus reducing the operation of video processing platforms and the efficiency of video processing. Summary of the Invention

[0003] The purpose of this application is to provide a scheduling method, apparatus, computer device, and computer-readable storage medium to solve the following problem: existing scheduling services lack flexibility, thereby reducing the operation of video processing platforms and the efficiency of video processing.

[0004] One aspect of this application provides a scheduling method for a DAG scheduling server, the method comprising:

[0005] Receive a call request, the call request including request parameters indicating the call triggering method;

[0006] The target DAG is obtained according to the scheduling request, and the target DAG includes multiple processing units with dependencies.

[0007] Target scheduling is performed according to the call triggering method and the target DAG to execute the processing units to be executed among the multiple units through the executor.

[0008] Optionally, the step of performing target scheduling based on the call triggering method and the target DAG includes:

[0009] The processing unit to be executed is determined based on the target DAG, and the unit state of the processing unit to be executed is set or updated; and

[0010] If the unit status setting or update of the processing unit to be executed is completed, the executor is notified to execute the processing unit to be executed.

[0011] Optionally, the triggering method is resource addition; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit state of the processing unit to be executed includes:

[0012] Identify the processing units in the target DAG with an in-degree of 0, and designate these processing units with an in-degree of 0 as the processing units to be executed; and

[0013] The DAG state server is requested to update the unit status of the processing unit to be executed to the ready state.

[0014] Optionally, the triggering method is resource update; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit state of the processing unit to be executed includes:

[0015] The processing unit to be executed is determined based on the target DAG; and

[0016] The system requests the DAG state server to update the unit status of the processing unit to be executed to the ready state, and also requests the DAG state server to update the unit status of each processing unit downstream of the processing unit to the registered state.

[0017] Optionally, the triggering method is resource deletion; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit state of the processing unit to be executed includes:

[0018] Each processing unit of the target DAG is determined as the processing unit to be executed; and

[0019] The DAG state server is requested to update the unit status of the processing unit to be executed to the ready state.

[0020] Optionally, the triggering method is a resource callback; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit state of the processing unit to be executed includes:

[0021] Based on the target DAG and the resource callback, determine the processing unit to be executed; and

[0022] The DAG state server is requested to update the unit status of the processing unit to be executed to the ready state.

[0023] Optionally, the method further includes:

[0024] Obtain the unit status of the processing unit to be executed from the DAG state server to determine whether to invoke it.

[0025] Optionally, the triggering method is completed by a processing unit; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit state of the processing unit to be executed includes:

[0026] Based on the target DAG and the completed processing units corresponding to the processing unit, the downstream processing units of the completed processing units are determined as the processing units to be executed; and

[0027] The DAG state server is requested to update the unit status of the processing unit to be executed to either a ready state or a waiting state.

[0028] Optionally, the triggering method is to re-execute the target DAG; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit state of the processing unit to be executed includes:

[0029] Identify the processing units in the target DAG with an in-degree of 0, and designate these processing units with an in-degree of 0 as the processing units to be executed; and

[0030] The system requests the DAG state server to update the unit status of the processing unit to be executed to the ready state, and requests the DAG state server to update the other processing units in the target DAG, excluding the processing unit to be executed, to the registered state.

[0031] Optionally, the triggering method is to re-execute the specified processing unit; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit state of the processing unit to be executed includes:

[0032] The specified processing unit in the target DAG is determined as the processing unit to be executed; and

[0033] The system requests the DAG state server to update the unit status of the processing unit to be executed to the ready state, and also requests the DAG state server to update the downstream processing units of the processing unit to be executed in the target DAG to the registered state.

[0034] Optionally, the processing unit to be executed corresponds to standard mode, synchronous mode, or asynchronous mode:

[0035] The processing unit corresponding to the standard mode includes a synchronous processing part and an asynchronous processing part;

[0036] The processing unit corresponding to the synchronous mode includes a synchronization processing part, which is set to a ready state when called;

[0037] The processing unit corresponding to the asynchronous mode includes an asynchronous processing part, which is set to a waiting state when called until it is set to a ready state after receiving a scheduling message.

[0038] One aspect of this application provides a DAG scheduling server, the apparatus comprising:

[0039] The first acquisition module is used to acquire a call request, wherein the call request includes request parameters indicating the call triggering method;

[0040] The second acquisition module is used to acquire the target DAG according to the scheduling request, wherein the target DAG includes multiple processing units with dependencies.

[0041] The execution module is used to perform target scheduling according to the call triggering method and the target DAG, so as to execute the processing unit to be executed among the multiple units through the executor.

[0042] One aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0043] One aspect of this application provides a computer-readable storage medium, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0044] The scheduling method, apparatus, computer equipment, and computer-readable storage medium provided in the embodiments of this application perform different scheduling based on the call triggering method and the target DAG, providing flexibility in scheduling services to improve the operation of the video processing platform and the efficiency of video processing. Attached Figure Description

[0045] Figure 1 A schematic diagram illustrating the system architecture of a scheduling system according to an embodiment of this application is shown.

[0046] Figure 2 This diagram schematically illustrates the operation process of a scheduling system according to an embodiment of this application.

[0047] Figure 3 This illustration schematically shows another operational process diagram of a scheduling system according to an embodiment of this application;

[0048] Figure 4 The diagram schematically illustrates the internal structure of the processing unit corresponding to the standard mode;

[0049] Figure 5 The diagram illustrates the state transition of a processing unit corresponding to the standard mode.

[0050] Figure 6 The diagram schematically illustrates the internal structure of the processing unit corresponding to the asynchronous mode;

[0051] Figure 7The diagram illustrates the state transition of the processing unit corresponding to the asynchronous mode.

[0052] Figure 8 The diagram schematically illustrates the internal structure of the processing unit corresponding to the synchronous mode;

[0053] Figure 9 The diagram illustrates the state transition of the processing unit corresponding to the synchronous mode.

[0054] Figure 10 A schematic diagram of a DAG is shown.

[0055] Figures 11(A) to (E) schematically illustrate the scheduling flowchart of the DAG scheduling server in automatic scheduling mode;

[0056] Figures 12(A) to (B) schematically illustrate the scheduling flowchart of the DAG scheduling server in passive scheduling mode;

[0057] Figure 13 This schematically illustrates a video DAG of a video processing scheme;

[0058] Figure 14 The flowchart illustrating the processing of the video submission unit in the video DAG is shown in the diagram.

[0059] Figure 15 The schematic diagram illustrates the processing flow of the video one-turn unit in the video DAG;

[0060] Figure 16 The flowchart illustrating the processing of the video review unit in the video DAG is shown in the diagram.

[0061] Figure 17 The schematic diagram illustrates the processing flow of the video conversion unit in the video DAG;

[0062] Figure 18 The schematic diagram illustrates the processing flow of the video distribution unit in the video DAG;

[0063] Figure 19 The flowchart illustrating the processing of video open units in a video DAG is shown in the diagram.

[0064] Figure 20 A flowchart illustrating the manuscript processing is shown in the diagram.

[0065] Figure 21 This diagram illustrates another processing flow for the manuscript;

[0066] Figure 22 A flowchart illustrating the scheduling method according to Embodiment 1 of this application is shown schematically;

[0067] Figure 23 for Figure 22 Flowchart of the sub-steps in step S2204;

[0068] Figures 24-30 for Figure 23 Flowchart of the sub-steps in step S2300;

[0069] Figure 31 A block diagram of a scheduling device according to Embodiment 2 of this application is schematically shown; and

[0070] Figure 32 The illustration shows a schematic diagram of the hardware architecture of a computer device suitable for implementing the scheduling method according to Embodiment 3 of this application. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0072] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0073] The inventors have learned that video processing primarily uses Databus messages and HTTP / RPC interfaces for communication, with business logic distributed across different servers. The most crucial aspects of video processing are transcoding and review, which involve numerous messages and complex processes. Developing complex, multi-process business logic often requires multiple interactions with various stakeholders, resulting in fragmentation, complexity, and difficulty in control.

[0074] This application aims to provide a scheduling system that processes data according to a Directed Acyclic Graph (DAG), orderly integrating business logic scattered across different services into a processing unit, which is then uniformly processed by the content platform's monitoring and scheduling services, thereby realizing the transformation of scattered logic into a unified scheduling mechanism.

[0075] The following is a definition of the terminology used in this application:

[0076] A directed acyclic graph (DAG) is a graph in which all edges point in the same direction, either from downstream to upstream or vice versa. Generally, for ease of observation, edges in a DAG are uniformly arranged to point from upstream to downstream to represent the dependencies between processing units. These dependencies are directional, and the direction of the edges indicates the directionality of the dependency. For example, if there is an edge between processing unit A and processing unit B pointing from A to B, then processing unit B depends on processing unit A.

[0077] An archive is a collection of multiple videos.

[0078] In-degree is an important concept in graph theory algorithms. It usually refers to the sum of the number of times a vertex in a directed graph is the endpoint of an edge in the graph.

[0079] Figure 1 The system architecture of the content processing scheduling system of this application is illustrated. The scheduling system is designed to produce and process different types of resources from various business units to meet increasingly diverse resource demands. The system can provide a unified processing solution and management capabilities for complex process sequences for various businesses, including videos, columns, music videos, comics, and music, facilitating the rapid development of new businesses and the upgrading and iteration of existing businesses. In this scheduling system, business logic can be abstracted into a Directed Acyclic Graph (DAG) to orderly integrate dispersed business logic into processing units. The DAG is used to organize the data processing flow, achieving abstract management of logical dependencies. That is, the node relationships in the DAG are extracted according to the processing order of resources to reduce business development costs and improve management / processing efficiency, thereby reducing business development costs and converging the resource processing process.

[0080] The scheduling system includes: a listening server 10, a DAG scheduling server 20, a DAG management server 30, a DAG status server 40, and an executor 50.

[0081] (i) Listening server 10, listens for and receives messages from inside and outside the system.

[0082] The listening server 10 can receive scheduling messages from external business modules, such as those provided through the submission event message queue and the video cloud message queue.

[0083] The listening server 10 can also receive messages provided by the scheduling system, such as unified processing of resource status change messages and callback messages, and initiate scheduling requests after parsing and verifying the messages. For example, if a resource changes, the resource module will generate a resource change and transmit the generated resource change to the listening server 10 through the data synchronization system (content event message queue).

[0084] (ii) DAG scheduling server 20, used to determine the processing unit to be executed based on DAG and unit status.

[0085] A DAG consists of multiple processing units and processing sequences, which represent the execution order and dependencies of each processing unit in the DAG.

[0086] The DAG scheduling server 20 can trigger scheduling based on a scheduling request provided by the listening server 10. For example, if a resource changes, the resource module generates a resource change request and transmits it to the listening server 10. The listening server 10 generates a scheduling request based on the resource change and sends the request to the DAG scheduling server 20.

[0087] The DAG scheduling server 20 can request the DAG status server 40 to provide the unit status of the processing unit to be executed and set the unit status of the processing unit to be executed, and notify the executor 50 to execute the processing unit to be executed.

[0088] The scheduling can include automatic scheduling and manual assignment. Automatic scheduling is triggered automatically within the scheduling system, such as by adding, editing, deleting, or completing resource processing. Manual assignment is executed by an external content management unit, used for data repair in special circumstances.

[0089] (III) DAG Management Server 30, which is responsible for managing DAGs and processing units in DAGs, including operations such as creation, editing, and graphic organization.

[0090] DAGs can be stored in DAG DB (DAG Database), and processing units can be stored in Processor DB (Processor Database).

[0091] The DAG management server 30 can be used to create / edit processing units and DAGs. For example, processing units can be added to a DAG, and the relationships between processing units within the DAG can be changed. A DAG consists of multiple processing units (also called "nodes") and lines connecting these units. A processing unit is an abstraction of specific processing logic, and lines are an abstraction of the processing order between units. Processing units are divided into general processing units and custom processing units. General processing units can be used in multiple DAGs and define processing logic commonly used in various business processes, such as image transfer and resource submission. Custom processing units are developed by developers based on specific business needs, and each processing unit can have different functions. Each processing unit can correspond to multiple states. Each processing unit can include asynchronous processing logic and / or synchronous processing logic.

[0092] (iv) DAG state server 40, used to provide query, update and delete unit status.

[0093] The unit state indicates the current state of the processing unit.

[0094] For example, as shown in Table 1, the unit state may include, but is not limited to, the following states: Cancel, Register, Ready, Running, Complete, Waiting (Callback), Execution Failure, Execution Error, Callback Failure, Callback Error, Cancellation Failure, Cancellation Error.

[0095]

[0096] Table 1

[0097] Taking a certain processing unit A as an example: (1) After the resource is deleted, processing unit A is set to the canceled state, i.e., "cancelled"; (2) Registered, used as the default state for waiting for scheduling; (3) When scheduled to be executed, processing unit A is set to the ready state, i.e., "ready"; (4) Before the executor 50 starts execution, it sets processing unit A to the running state, i.e., "processing"; (5) When processing unit A finishes all execution, it is set to the completed state, i.e., "completed"; (6) If an expected problem occurs during execution, processing unit A is set to the execution failure state, i.e., "execution failure"; (7) If an unexpected problem occurs, processing unit A is... (8) When processing unit A waits for an external message, it will be set to a waiting state, i.e., "waiting (callback)"; (9) If an expected problem occurs during the callback, processing unit A will be set to a callback failure state, i.e., "callback failure"; (10) If an unexpected problem occurs, processing unit A will be set to a callback error state, i.e., "callback error"; (11) If an expected problem occurs during cancellation, processing unit A will be set to a cancellation failure state, i.e., "cancellation failure"; (12) If an unexpected problem occurs, processing unit A will be set to a cancellation error state, i.e., "cancellation error".

[0098] (v) Actuator 50, used to execute each processing unit.

[0099] The executor 50 receives the processing unit scheduling message transmitted by the DAG scheduling server 20 and performs processing operations according to the processing unit scheduling message. In order to ensure the orderliness of the processing units, the executor 50 is equipped with a processing unit scheduling message queue to ensure that the processing unit scheduling messages can be entered and exited in an orderly manner, thereby ensuring that the processing units to be executed perform their processing operations in an orderly manner.

[0100] Through various devices, unified monitoring, scheduling, DAG and processing unit management, and state management are achieved. This enables the scheduling system to efficiently and centrally handle various complex businesses and interactions, increasing management efficiency, reducing business development costs, and streamlining resource processing. Furthermore, by setting various states for processing units and their transitions, and through centralized management on the DAG server, the system effectively improves the utilization efficiency of processing units and reduces conflicts in their use.

[0101] The overall architecture of the scheduling system has been described above. Figure 2As shown, the scheduling process of the scheduling system generally includes the following stages: S200, the DAG scheduling server 20 obtains DAG information from the DAG management server 30 via RPC request; S202, the DAG scheduling server 20 obtains unit status information from the DAG status server 40 via RPC request; S204, the DAG scheduling server 20 calculates the next processing unit based on the dependencies between each processing unit in the DAG; S206, the DAG scheduling server 20 sets the unit status to ready state via RPC request; S208, the DAG scheduling server 20 notifies the executor 50 to execute the calculated next processing unit through the processing unit scheduling message queue (which can be an asynchronous queue).

[0102] After receiving a notification message from the processing unit scheduling message queue, executor 50 obtains the status of the processing unit from the DAG state server 40 and verifies whether it is in a ready state. If the verification is successful, the processing unit is set to a running state. Then, based on the notification message, it selects to execute the synchronous or asynchronous processing part of the processing unit. After execution, depending on the specific situation, it sends an RPC request to the DAG state server 40 to update the processing unit to a completed or failed state. The DAG state server 40 returns the update result. The listening server 10 listens for processing unit execution completion messages sent by the DAG state server 40 to perform scheduling loops.

[0103] The following is an operation flowchart of the scheduling system, as shown below. Figure 3 As shown, the operation process is as follows:

[0104] S300, listens for scheduling messages on server 10 to initiate a scheduling request, which may be an RPC request;

[0105] S302, DAG scheduling server 20 requests DAG information from DAG management server 30;

[0106] S304, DAG management server 30 returns DAG information to DAG scheduling server 20;

[0107] S306, DAG scheduling server 20 requests the unit status information of each processing unit in DAG from DAG status server 40;

[0108] S308, DAG status server 40 returns the unit status information of each processing unit in the DAG to DAG scheduling server 20;

[0109] S310, DAG scheduling server 20 calculates the next processing unit to be processed based on DAG information;

[0110] S312, DAG scheduling server 20 requests DAG status server 40 via RPC to set the unit status of the processing unit to ready state;

[0111] S314, DAG state server 40 sets the unit status of the processing unit to the ready state and returns information to DAG scheduling server 20;

[0112] S316, the DAG scheduling server 20 notifies the executor 50 to execute the processing unit through the message queue (processing unit message scheduling queue);

[0113] S318, Executor 50 retrieves the processing unit from the message queue in preparation for execution;

[0114] S320, the actuator 50 obtains the unit status of the processing unit from the DAG state server 40 via an RPC request;

[0115] S322, DAG state server 40 returns the unit status of the processing unit to executor 50;

[0116] S324, Actuator 50 verifies whether the unit state of the processing unit is ready;

[0117] S326, the actuator 50 requests the DAG status server 40 via RPC to set the unit status of the processing unit to the running state;

[0118] S328, DAG state server 40 sets the unit state of the processing unit to running state and returns information to executor 50;

[0119] S330, actuator 50 executes the synchronous or asynchronous processing part of the processing unit;

[0120] S332, based on the completion status of the processing unit, the actuator 50 requests the DAG status server 40 via RPC to set the unit status to either a completed or failed state;

[0121] S334, DAG state server 40 sets the unit status of the processing unit to completion or failure and returns information to executor 50;

[0122] S336, DAG state server 40 sends a completion message to listening server 10, so that listening server 10 can initiate the next scheduling request to achieve cyclic scheduling.

[0123] As can be seen from the above, the scheduling system involves processing units, state transitions of the processing units, DAGs, etc., which are described below with examples:

[0124] ① Regarding the processing unit and its state transition:

[0125] The processing unit may include synchronous processing, asynchronous processing, and cancellation processing. The applicant has found that different business scenarios have different requirements for the processing unit; for example, in some business scenarios, the processing unit needs to execute asynchronously, not synchronously. Therefore, the processing unit can be configured with multiple modes to adapt to different execution requirements. These three modes may include standard mode, asynchronous mode, and synchronous mode.

[0126] Firstly, the standard model:

[0127] like Figure 4 As shown, the processing unit can include synchronous and asynchronous processing parts. For the synchronous processing part, the processing unit is set to a ready state during scheduling and to a running state before execution, then begins execution, for example, starting a long-running task. Afterward, the processing unit is set to a waiting state. For the asynchronous processing part, when the external listening service receives a task completion message, the DAG scheduling server 20 initiates scheduling. The processing unit is also set to both a ready and running state during scheduling. After completing the asynchronous processing part, the processing unit is set to a completed state, ending the task for this processing unit.

[0128] like Figure 5 As shown, the state transition of the processing unit in the standard mode is as follows:

[0129] State 1: When a processing unit is scheduled, it is first set to the "register" state;

[0130] State 2: When the processing unit is set to the "ready" state in the scheduler;

[0131] State 3: When the processing unit is ready to execute the synchronous processing part, it is set to the running state.

[0132] State 4: After the processing unit finishes execution, it is set to the "waiting" state.

[0133] If an expected situation occurs during the execution of the synchronization processing section, the processing unit is set to an execution failure state. Conversely, if an unexpected situation or an exception occurs in the scheduling service itself during the execution of the synchronization processing section, the processing unit is set to an execution error state.

[0134] State 5: After receiving the callback message, the processing unit will be set to the "ready" state again;

[0135] State 6: The processing unit is set to the "running" state before executing the asynchronous processing part, and is set to the "finish" state after the asynchronous processing is completed.

[0136] If an expected situation occurs during the execution of the asynchronous processing portion, the processing unit is set to a callback failure state. If an unexpected situation or an exception occurs in the scheduling service itself during the execution of the asynchronous processing portion, the processing unit is set to a callback error state.

[0137] Secondly, asynchronous mode:

[0138] Asynchronous mode is a simplified form of the standard mode and is suitable for business scenarios that only have callbacks. Figure 6 It is the internal structure of the processing unit in asynchronous mode. Figure 7 This is the state transition diagram for asynchronous processing units. When a processing unit is scheduled, it is first set to the registered state. If the DAG scheduling server 20 detects that the processing unit belongs to asynchronous mode, it directly sets it to the waiting state. After receiving a callback message, the processing unit is set to the ready state again, and before executing the asynchronous processing part, it is set to the running state, and after the asynchronous processing is completed, it is set to the completed state. If an expected situation occurs during the execution of the asynchronous processing part, the processing unit will be set to the callback failure state. If an unexpected situation occurs during the execution of the asynchronous processing part or an exception occurs in the scheduling service itself, the processing unit will be set to the callback error state.

[0139] Third, synchronization mode:

[0140] Synchronous mode is a simplified form of the standard mode, where the processing unit only has a synchronous processing part, and it is suitable for business scenarios without callbacks. Figure 8 It is the internal structure of the synchronous mode processing unit. Figure 9 This is the state transition diagram of the synchronous mode processing unit. When a processing unit is scheduled, it is first set to the registered state, and then to the ready state during scheduling. When preparing to execute the synchronous processing part, the processing unit is set to the running state, and after execution is completed, it is set to the completed state, ending the operation of this processing unit.

[0141] Without differentiating processing units by mode, each processing unit comprises three parts: synchronous processing, asynchronous processing, and cancellation processing. However, some processing units in certain services do not require synchronous processing. If the previous approach were followed, the DAG scheduling server 20 would still instruct the executor 50 to execute the synchronous processing part of the processing unit, setting the processing unit to a waiting state without executing any specific business code. This is equivalent to the DAG scheduling server 20 and the executor 50 performing an idle process. In this embodiment, dividing the processing units into the above three modes has the following advantages: If some processing units in a certain service do not require synchronous processing, the DAG scheduling server 20 can directly set that processing unit to a waiting state. Therefore, it saves one idle execution step by the executor 50, and also saves the database interaction caused by unit state verification during idle execution between the DAG scheduling server 20 and the executor 50, thereby improving the overall operation and processing efficiency of the scheduling system.

[0142] That is, by using synchronous and asynchronous modes, the processing unit can be more closely matched to specific business scenarios, more effectively save scheduling resources, reduce the workload of executor 50, reduce the number of interactions with the database, and effectively improve the operation and processing efficiency of the entire scheduling system.

[0143] ②Regarding DAG:

[0144] Figure 10 This is a schematic diagram of a Directed Acyclic Graph (DAG). The DAG includes processing units 1-6 and the dependencies and execution order between them. It should be noted that each processing unit automatically checks the status of all upstream processing units before execution; the current processing unit will only execute when all upstream processing units are in a completed state. Furthermore, each processing unit automatically attempts to trigger downstream processing units when it becomes completed. Different services and different logics correspond to different DAGs. During the operation of the scheduling system, the DAG scheduling server 20 can obtain the nodes (i.e., processing units) of the DAG using a breadth-first traversal.

[0145] Scheduling can be implemented based on the DAG and the unit states of the processing units within the DAG, i.e., determining which processing unit is to be executed (i.e., the next processing unit to be processed). This scheduling can be triggered based on resource changes and the completion of a processing unit's execution. The triggering methods can include the following:

[0146] The first method: resource addition;

[0147] The second method: resource updates;

[0148] The third method: resource deletion;

[0149] The fourth type: resource callback;

[0150] The fifth type: The processing unit changes from the running state to the completed state.

[0151] Listening server 10 sends different request parameters to the RPC scheduling interface of DAG scheduling server 20 according to different triggering methods. The triggering method is determined by the scheduling type field. DAG scheduling server 20 selects different scheduling strategies according to different request parameters.

[0152] The scheduling can include automatic scheduling mode and passive scheduling mode.

[0153] The following are examples of the scheduling process in several automatic scheduling modes:

[0154] As shown in Figure 11(A), when the scheduling trigger method is resource addition, the DAG scheduling server 20 can perform the following operations: Step S1100A, obtain the DAG of the corresponding resource; Step S1102A, calculate all processing units with an in-degree of 0 in the DAG; Step S1104A, set the processing units with an in-degree of 0 to the ready state; Step S1106A, notify the executor 50 to execute the synchronous processing part of all processing units with an in-degree of 0. It should be noted that there is no "obtaining unit status" step in this scheduling process.

[0155] As shown in Figure 11(B), when the scheduling trigger method is resource update, the scheduling server 20 can perform the following operations: Step S1100B, obtain the DAG of the corresponding resource; Step S1102B, determine the processing unit to be executed according to the DAG; Step S1104B, obtain the unit status of the processing unit to be executed (used to verify whether it can be called); Step S1106B, set the processing unit to be executed to the ready state, and reset all processing units downstream of the processing unit to be executed to the registered state to wait for subsequent scheduling; Step S1108B, notify the executor 50 to execute the synchronous processing part of the processing unit to be executed.

[0156] As shown in Figure 11(C), when the scheduling trigger method is resource deletion, the DAG scheduling server 20 can perform the following operations: Step S1100C, obtain the DAG of the corresponding resource; Step S1102C, calculate all processing units in the DAG; Step S1104C, set all processing units in the DAG to the ready state; Step S1106C, notify the executor 50 to execute the cancellation processing part of each processing unit in the DAG. It should be noted that there is no "obtaining unit status" step in this scheduling process.

[0157] As shown in Figure 11(D), when the scheduling trigger method is resource callback, the DAG scheduling server 20 can perform the following operations: Step S1100D, obtain the DAG of the corresponding resource; Step S1102D, find the processing unit to be executed according to the DAG and the callback message; Step S1102D, obtain the unit status of the processing unit to be executed (used to verify whether it can be called); Step S1106D, set the processing unit to the ready state; Step S1108D, notify the executor 50 to execute the asynchronous processing part of the processing unit.

[0158] As shown in Figure 11(E), when the scheduling trigger is a completion message of a processing unit, the DAG scheduling server 20 can perform the following operations: Step 1100E, obtain the DAG of the corresponding resource; Step 1102E, find the downstream processing unit of the completed processing unit according to the DAG; Step 1104E, set the downstream processing unit to a ready state or a waiting state; Step 1106E, notify the executor 50 to execute the asynchronous processing part of the downstream processing unit. It should be noted that there is no "obtaining unit status" step in this scheduling process.

[0159] The above describes automatic scheduling. Automatic scheduling is triggered automatically based on the addition, update, deletion of resources, and callback messages. However, during business development iterations, special situations such as data washing may occur, requiring the re-execution of processing units. To adapt to these situations, the scheduling system extends automatic scheduling with a manual assignment method to manually execute processing units.

[0160] The following are examples of scheduling processes under several passive scheduling modes (manual scheduling modes):

[0161] As shown in Figure 12(A), if it is necessary to re-execute all processing units in the DAG, the DAG scheduling server 20 performs the following operations: Step S1200A, obtain the DAG of the current resource; Step S1202A, calculate the processing units with an in-degree of 0 based on the DAG; Step S1204A, set the processing units with an in-degree of 0 to the ready state, and set the other processing units in the DAG to the registered state; Step S1206A, notify the executor 50 to start execution. It should be noted that the "obtaining unit status" step may be omitted in this scheduling process.

[0162] As shown in Figure 12(B), if a specific processing unit in the DAG needs to be re-executed, the DAG scheduling server 20 performs the following operations: Step S1200B, obtain the DAG of the current resource; Step S1202B, calculate all downstream processing units of the specified processing unit according to the breadth-first traversal of the DAG; Step S1204B, set the specified processing unit to the ready state and set the downstream processing units to the registered state; Step S1206B, notify the executor 50 to start execution. It should be noted that the "obtaining unit status" step may be omitted in this scheduling process.

[0163] The above scheduling process effectively expands the capabilities of the scheduling system, enabling it to effectively address special business needs and making the system more flexible in areas such as data rewashing and anomaly correction.

[0164] The scheduling system can be applied in various scenarios, such as video processing. The following section will introduce a video processing scheme based on the scheduling system.

[0165] In this embodiment, the video processing scheme can be configured as follows: Figure 13 The video DAG shown is a video DAG. This video DAG includes multiple processing units, and the dependencies between these processing units are as follows:

[0166] Video submission unit A1 is the starting node, video conversion unit A2 is the downstream node of video submission unit A1, video review unit A3 and video conversion unit A4 are parallel and both are downstream nodes of video conversion unit A2, video distribution unit A5 is the downstream node of video conversion unit A4, and video opening unit A6 is the downstream node of video review unit A3 and video distribution unit A5. In this embodiment, the video DAG can create video review tasks corresponding to video review unit A3 and high-definition transcoding tasks corresponding to video conversion unit A4 in parallel after video conversion unit A2, and aggregate video review unit A3 and video distribution unit A5 through video opening unit A6. When video review unit A3 and video distribution unit A5 are both completed, video opening unit A6 is triggered to enter the running state to open the video. The above parallel result can improve the video processing speed and enhance the user experience. According to statistics, using the above parallel setting, the average video processing time decreased from 610 seconds to 530 seconds, a decrease of 13%.

[0167] Video submission unit A1 is a synchronous processing unit used to create video resources and submit them to the video cloud transcoding platform to initiate transcoding.

[0168] like Figure 14As shown, users upload videos to the content resource end through the submission service, and the data synchronization system provides the video addition message from the content resource end to the listening server 10. In response to the video addition message, the listening server 10 sends a scheduling request to the DAG scheduling server 20. In response to the scheduling request, the DAG scheduling server 20 notifies the executor 50 through the queue to execute the video submission unit A1: (1) Prepare for execution: Request the DAG status server 40 to update the unit status of the video processing unit A1 to the running status; (2) Execute: Call the interface to publish the video submission message to notify the video cloud transcoding platform to start the transcoding service; (3) Execution complete: Request the DAG status server 40 to update the unit status of the video processing unit A1 to the completed status. The DAG status server 40 generates a "video submission complete message" and submits the "video submission complete message" to the listening server 10.

[0169] Video conversion unit A2 is an asynchronous processing unit. Video conversion is implemented within a video cloud transcoding platform, where user-uploaded videos are first transcoded into lower-resolution versions. These lower-resolution videos can then be used in the video content review process, resulting in a fast and time-efficient process.

[0170] like Figure 15 As shown, when video conversion unit A2 is scheduled, it is set to the registered state. If the DAG scheduling server 20 detects that video conversion unit A2 is in asynchronous mode, it will directly set it to the waiting state. After receiving the callback message ("video conversion success message") from the video cloud transcoding platform, video conversion unit A2 will be set to the ready state again and the executor 50 will be notified through the queue to perform the following operations: (1) Prepare for execution: set the unit state of video conversion unit A2 to the running state; (2) Execute: call the interface to update the video information according to the callback message; (3) Execution complete: request the DAG state server 40 to update the unit state of video conversion unit A2 to the completed state. It should be noted that after setting the unit state of video conversion unit A2 to the completed state, the DAG state server 40 generates a "video conversion completion message" and submits the "video conversion completion message" to the listening server 10.

[0171] Video review unit A3 is a standard processing unit used to create video review tasks on the review platform. The review platform sends the review results to video review unit A3 via an asynchronous message queue. The asynchronous processing part of video review unit A3 receives the review results. The review results include "approved" and "disapproved." If the review is unsuccessful, the video needs to be re-edited by the user.

[0172] like Figure 16As shown, the listening server 10 receives a "video one-way completion message". In response to the "video one-way completion message", the listening server 10 sends a scheduling request to the DAG scheduling server 20. In response to the scheduling request, the DAG scheduling server 20 notifies the executor 50 to execute the video review unit A3 through a queue. The video review unit A3 includes a synchronous processing part and an asynchronous processing part. The synchronous processing part includes: (1) preparing for execution: requesting the DAG status server 40 to update the unit status of the video review unit A3 to the running state; (2) executing: calling the interface to create a video review task on the review platform; (3) completing execution: requesting the DAG status server 40 to update the unit status of the video review unit A3 to the waiting state. The asynchronous processing part is callback processing, including: after receiving a callback message ("review message") from the review platform, the video review unit A3 will be set to the ready state again, and the executor 50 will be notified through the queue to perform the following operations: (1) Prepare for execution: set the unit state of the video review unit A3 to the running state; (2) Execute: update the review status of the video according to the callback message from the review platform; (3) Execution complete: request the DAG state server 40 to update the unit state of the video review unit A3 to the completed state. After being updated to the completed state, the DAG state server 40 generates a "video review completed message" and submits the "video review completed message" to the listening server 10.

[0173] Video conversion unit A4 is an asynchronous processing unit. Video conversion is implemented in a video cloud transcoding platform, whereby the user-uploaded video is first transcoded into a higher-resolution video. This higher-resolution video can be used for user-facing videos and has a unified format.

[0174] like Figure 17 As shown, when video conversion unit A4 is scheduled, it is set to the registered state. If the DAG scheduling server 20 detects that video conversion unit A4 is in asynchronous mode, it will directly set it to the waiting state. After receiving the callback message ("video conversion message") from the video cloud transcoding platform, video conversion unit A4 will be set to the ready state again and the executor 50 will be notified through the queue to perform the following operations: (1) Prepare for execution: Set the unit state of video conversion unit A4 to the running state; (2) Execute: Call the interface to update the video information according to the callback message returned by the video cloud transcoding platform; (3) Execution complete: Request the DAG state server 40 to update the unit state of video conversion unit A4 to the completed state. It should be noted that after updating to the completed state, the DAG state server 40 generates a "video conversion completed message" and submits the "video conversion completed message" to the listening server 10.

[0175] After the video cloud transcoding platform performs secondary transcoding on the video, it distributes the transcoded video to a CDN (Content Delivery Network) so that users can quickly access the video. Video distribution unit A5 receives a distribution completion message after distribution is finished.

[0176] Video distribution unit A5 is an asynchronous processing unit.

[0177] like Figure 18 As shown, when video distribution unit A5 is scheduled, it is set to the registered state. If the DAG scheduling server 20 detects that video distribution unit A5 is in asynchronous mode, it will directly set it to the waiting state. After receiving the callback message ("video distribution message") from the video cloud transcoding platform, video distribution unit A5 will be set to the ready state again and will notify executor 50 to perform the following operations through the queue: (1) Prepare for execution: set the unit state of video transcoding unit A4 to the running state; (2) Execute: call the interface video information according to the distribution message from the video cloud transcoding platform; (3) Execution complete: request DAG state server 40 to update the unit state of video distribution unit A5 to the completed state. After updating to the completed state, DAG state server 40 generates a "video distribution completed message" and submits the "video distribution completed message" to the listening server 10.

[0178] Video open unit A6 is a processing unit for synchronous mode.

[0179] like Figure 19 As shown, the listening server 10 receives the "Video Second-Transfer Completed Message" and the "Video Review Completed Message". In response to the "Video Second-Transfer Completed Message" and the "Video Review Completed Message", the listening server 10 sends a scheduling request to the DAG scheduling server 20. In response to the scheduling request, the DAG scheduling server 20 sets the video open unit A6 to the ready state and notifies the executor 50 to execute through the queue: (1) Prepare for execution: set the unit status of the video open unit A6 to the running state (processing); (2) Execute: call the interface to update the video information and mark the video as open; (3) Execution complete: request the DAG state server 40 to update the unit status of the video open unit A6 to the completed state. After updating to the completed state, the DAG state server 40 generates a "Video Opening Completed Message" and submits the "Video Distribution Completed Message" to the listening server 10, and the scheduling ends.

[0180] The video processing solution has been introduced above. The manuscript processing solution, which is a collection of multiple videos, will be introduced below.

[0181] The manuscript DAG can include the following two processing units: manuscript submission unit and manuscript aggregation unit.

[0182] The manuscript submission unit means that when a user submits a manuscript, corresponding actions such as notifying downstream submission events and recording submission logs will be executed.

[0183] The article aggregation unit determines the status of an article resource based on the status of all video resources under that resource. Once all videos under an article resource have been processed, the article aggregation unit will set the article to an open state and make it available to users.

[0184] In this embodiment, to enrich the relationships between resources, the scheduling system provides content relationship services, such as parent-child / subordinate, sibling-copy, etc. For example, the relationship between a manuscript and a video is subordinate. Establishing relationships between resources allows associating the processing units of their respective resources, thus enabling linkage between DAGs. Taking manuscripts and videos as an example, a manuscript can include multiple videos. In the calling system, the manuscript is the parent resource, and the video is the child resource. The processing of the video DAG triggers the processing of the manuscript DAG.

[0185] The following is combined with Figure 20 This section describes the processing flow of a manuscript by the scheduling system.

[0186] This document includes a first video and a second video. (For example...) Figure 20 As shown, the scheduling system creates two video processing resources and one manuscript processing resource. These three resources can be associated through a parent-child relationship, with the manuscript serving as the parent resource for the two videos and the videos serving as the child resources for the manuscript.

[0187] The manuscript corresponds to the manuscript DAG, which includes the manuscript submission unit C1 and the manuscript aggregation unit C2.

[0188] The first video corresponds to the first video DAG, which includes a video submission unit A1, a video conversion unit A2, a video review unit A3, a video conversion unit A4, a video distribution unit A5, and a video opening unit A6.

[0189] The second video corresponds to the second video DAG, which includes a video submission unit B1, a video first conversion unit B2, a video review unit B3, a video second conversion unit B4, a video distribution unit B5, and a video opening unit B6.

[0190] Taking the first video DAG as an example, video submission unit A1 triggers the video transcoding operation (video transcoding 1) on the video cloud transcoding platform. Video transcoding 1 unit A2 is an asynchronous processing unit. After video submission unit A1 completes, video transcoding 1 unit A2 is set to a waiting state, waiting to receive a success or failure message for low-definition video transcoding from the video cloud transcoding platform. If transcoding is successful, video transcoding 1 unit A2 is set to a completed state. The scheduling system determines the next processing unit based on the first video DAG, namely video review unit A3 and video transcoding 2 unit A4. Video review unit A3 and video transcoding 2 unit A4 are executed in parallel. Video review unit A3 is a standard processing unit. In the synchronous processing part, video review unit A3 creates a video review task on the review platform and waits for review messages through the asynchronous processing part. Video transcoding 2 unit A4 is also an asynchronous processing unit, which is set to a waiting state and waits to receive a success or failure message for high-definition transcoding. Video distribution unit A5 is also an asynchronous processing unit. After the video cloud transcoding platform distributes the video to the CDN, video distribution unit A5 will receive a distribution completion message. Before scheduling video opening unit A6, the scheduling system checks whether the video review and video distribution processing units have been completed. If both have been completed, the scheduling system will schedule video opening unit A6. Video opening unit A6 is a synchronous processing unit. Video opening unit A6 sets the video resource status to open during synchronous execution.

[0191] The manuscript DAG is associated with the first video DAG and the second video DAG. This association can include: association processing, linkage timing, and linkage results. Within the manuscript DAG, the manuscript submission unit C1 is a synchronous processing unit. Manuscript submission unit C1 notifies other business parties of new manuscript arrivals and records manuscript submission logs, etc. The manuscript aggregation unit C2 is an asynchronous processing unit; its asynchronous processing part receives linkage operations from the first video DAG and the second video DAG.

[0192] Video review units A3 / B3 and video opening units A6 / B6 trigger document aggregation unit C2. For example, if the first video is rejected, video review unit A3 generates a callback message to notify document aggregation unit C2. Upon receiving the callback message indicating rejection, document aggregation unit C2 directly rejects the document and notifies the user, without waiting for processing of other videos (second videos). If the first video passes review, video opening unit A6 notifies document aggregation unit C2 via a callback message. After receiving the approval message, document aggregation unit C2 finds all video sub-resources based on content relationships and then checks the status of all videos. If other videos (second videos) are pending review, document aggregation unit C2 continues to wait. If both the first and second videos are open, the document will be opened, and a document opening notification will be sent to the user.

[0193] The above describes the joint processing of manuscripts and videos. The following section discusses special review cases:

[0194] For example, if the first video is rejected during review, the video review unit A3 will generate a callback message to notify the document aggregation unit. Upon receiving the rejection message, the document aggregation unit will directly reject the document and notify the user. Meanwhile, the other videos (the second video) will continue processing until they enter the open state. After modification, if the first video passes review, the video opening unit A6 will notify the document aggregation unit via a callback message. Upon receiving the approval message, the document aggregation unit will locate all video sub-resources based on content relationships and then check the status of all videos. If the other videos (the second video) pass review, and both the first video and the other videos (the second video) are in the open state, the document will be opened, and an open notification will be sent to the user. Figure 21 As shown, the first video undergoes two video processing cycles, while the second video undergoes one video processing cycle.

[0195] Example 1

[0196] This embodiment describes the execution entity using the DAG scheduling server 20 as the main body. For technical details and effects, please refer to the above.

[0197] Figure 22 A flowchart illustrating a scheduling method according to Embodiment 1 of this application is shown schematically. Figure 22 As shown, the scheduling method may include steps S2300 to S2304, wherein:

[0198] Step S2200: Receive a call request, the call request including request parameters for indicating the call triggering method.

[0199] Step S2202: Obtain the target DAG according to the scheduling request. The target DAG includes multiple processing units with dependencies.

[0200] Step S2204: Target scheduling is performed according to the call triggering method and the target DAG to execute the processing unit to be executed among the multiple units through the executor.

[0201] As an example, such as Figure 23 As shown, step S2204 may include:

[0202] Step S2300: Determine the processing unit to be executed based on the target DAG, and set or update the unit state of the processing unit to be executed; and

[0203] Step S2302: If the unit status setting or update of the processing unit to be executed is completed, then the executor is notified to execute the processing unit to be executed.

[0204] As an example, the triggering method is resource addition. Figure 24 As shown, step S2300 may include:

[0205] Step S2400: Determine the processing unit with an in-degree of 0 in the target DAG, and designate the processing unit with an in-degree of 0 as the processing unit to be executed; and

[0206] Step S2402: Request the DAG state server to update the unit state of the processing unit to be executed to the ready state.

[0207] As an example, the triggering method is a resource update. Figure 25 As shown, step S2300 may include:

[0208] Step S2500: Determine the processing unit to be executed based on the target DAG; and

[0209] Step S2502: Request the DAG state server to update the unit status of the processing unit to be executed to the ready state, and request the DAG state server to update the unit status of each processing unit downstream of the processing unit to be executed to the registered state.

[0210] As an example, the triggering method is resource deletion. Figure 26 As shown, step S2300 may include:

[0211] Step S2600: Determine each processing unit of the target DAG as the processing unit to be executed; and

[0212] Step S2602: Request the DAG state server to update the unit state of the processing unit to be executed to the ready state.

[0213] As an example, the triggering method is a resource callback. Figure 27 As shown, step S2300 may include:

[0214] Step S2700: Determine the processing unit to be executed based on the target DAG and the resource callback; and

[0215] Step S2702: Request the DAG state server to update the unit state of the processing unit to be executed to the ready state.

[0216] As an example, the scheduling method further includes: obtaining the unit status of the processing unit to be executed from the DAG state server to determine whether to call it.

[0217] As an example, the triggering method is completed by the processing unit. For example... Figure 28 As shown, step S2300 may include:

[0218] Step S2800: Based on the target DAG and the completed processing units corresponding to the processing unit, determine the downstream processing units of the completed processing units as the processing units to be executed; and

[0219] Step S2802: Request the DAG state server to update the unit state of the processing unit to be executed to a ready state or a waiting state.

[0220] As an example, the triggering method is to re-execute the target DAG. For example... Figure 29 As shown, step S2300 may include:

[0221] Step S2900: Determine the processing units with an in-degree of 0 in the target DAG, and designate the processing units with an in-degree of 0 as the processing units to be executed; and

[0222] Step S2902: Request the DAG state server to update the unit status of the processing unit to be executed to the ready state, and request the DAG state server to update the other processing units in the target DAG, excluding the processing unit to be executed, to the registered state.

[0223] As an example, the triggering method is to re-execute the specified processing unit. For example... Figure 30 As shown, step S2300 may include:

[0224] Step S3000: Determine the specified processing unit in the target DAG as the processing unit to be executed;

[0225] Step S3002: Request the DAG state server to update the unit status of the processing unit to be executed to the ready state, and request the DAG state server to update the downstream processing unit of the processing unit to be executed in the target DAG to the registered state.

[0226] As an example, the processing unit to be executed corresponds to standard mode, synchronous mode, or asynchronous mode:

[0227] The processing unit corresponding to the standard mode includes a synchronous processing part and an asynchronous processing part;

[0228] The processing unit corresponding to the synchronous mode includes a synchronization processing part, which is set to a ready state when called;

[0229] The processing unit corresponding to the asynchronous mode includes an asynchronous processing part, which is set to a waiting state when called until it is set to a ready state after receiving a scheduling message.

[0230] Example 2

[0231] Figure 31 The diagram schematically illustrates a scheduling device according to Embodiment 2 of this application. This scheduling device can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program modules referred to in the embodiments of this application are a series of computer program instruction segments capable of performing specific functions. The following description will specifically introduce the functions of each program module in this embodiment.

[0232] like Figure 31 As shown, the scheduling device 3100 may include a first acquisition module 3110, a second acquisition module 3120, and an execution module 3130, wherein:

[0233] The first acquisition module 3110 is used to acquire a call request, the call request including request parameters indicating the call triggering method;

[0234] The second acquisition module 3120 is used to acquire a target DAG according to the scheduling request, wherein the target DAG includes multiple processing units with dependencies.

[0235] The execution module 3130 is used to perform target scheduling according to the call triggering method and the target DAG, so as to execute the processing unit to be executed among the multiple units through the executor.

[0236] As an example, the triggering methods of the scheduling message include resource addition, resource update, resource deletion, resource callback, processing unit completion, re-execution of the target DAG and / or re-execution of the specified processing unit; the execution module 3130 is further used for:

[0237] The processing unit to be executed is determined based on the target DAG, and the unit state of the processing unit to be executed is set or updated; and

[0238] If the unit status setting or update of the processing unit to be executed is completed, the executor is notified to execute the processing unit to be executed.

[0239] As an example, the triggering method is the addition of the resource; the execution module 3130 is also used for:

[0240] Identify the processing units in the target DAG with an in-degree of 0, and designate these processing units with an in-degree of 0 as the processing units to be executed; and

[0241] The DAG state server is requested to update the unit status of the processing unit to be executed to the ready state.

[0242] As an example, the triggering method is the resource update; the execution module 3130 is also used for:

[0243] The processing unit to be executed is determined based on the target DAG; and

[0244] The system requests the DAG state server to update the unit status of the processing unit to be executed to the ready state, and also requests the DAG state server to update the unit status of each processing unit downstream of the processing unit to the registered state.

[0245] As an example, the triggering method is the deletion of the resource; the execution module 3130 is also used for:

[0246] Each processing unit of the target DAG is determined as the processing unit to be executed; and

[0247] The DAG state server is requested to update the unit status of the processing unit to be executed to the ready state.

[0248] As an example, the triggering method is the resource callback; the execution module 3130 is also used for:

[0249] Based on the target DAG and the resource callback, determine the processing unit to be executed; and

[0250] The DAG state server is requested to update the unit status of the processing unit to be executed to the ready state.

[0251] As an example, the execution module 3130 is further configured to: obtain the unit status of the processing unit to be executed from the DAG state server in order to determine whether to call it.

[0252] As an example, the triggering method is completed by the processing unit; the execution module 3130 is also used for:

[0253] Based on the target DAG and the completed processing units corresponding to the processing unit, the downstream processing units of the completed processing units are determined as the processing units to be executed; and

[0254] The DAG state server is requested to update the unit status of the processing unit to be executed to either a ready state or a waiting state.

[0255] As an example, the triggering method is to re-execute the target DAG; the execution module 3130 is also used for:

[0256] Identify the processing units in the target DAG with an in-degree of 0, and designate these processing units with an in-degree of 0 as the processing units to be executed; and

[0257] The system requests the DAG state server to update the unit status of the processing unit to be executed to the ready state, and requests the DAG state server to update the other processing units in the target DAG, excluding the processing unit to be executed, to the registered state.

[0258] As an example, the triggering method is the re-execution of the specified processing unit; the execution module 3130 is further configured to:

[0259] The specified processing unit in the target DAG is determined as the processing unit to be executed; and

[0260] The system requests the DAG state server to update the unit status of the processing unit to be executed to the ready state, and also requests the DAG state server to update the downstream processing units of the processing unit to be executed in the target DAG to the registered state.

[0261] As an example, the processing unit to be executed corresponds to standard mode, synchronous mode, or asynchronous mode:

[0262] The processing unit corresponding to the standard mode includes a synchronous processing part and an asynchronous processing part;

[0263] The processing unit corresponding to the synchronous mode includes a synchronization processing part, which is set to a ready state when called;

[0264] The processing unit corresponding to the asynchronous mode includes an asynchronous processing part, which is set to a waiting state when called until it is set to a ready state after receiving a scheduling message.

[0265] Example 3

[0266] Figure 32 This illustration schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing a scheduling method according to Embodiment 3 of this application. In this embodiment, the computer device 10000 can serve as a listening server 10, a DAG scheduling server 20, a DAG management server 30, a DAG state server 40, an actuator 50, or as part of any of these devices. In this embodiment, the computer device 10000 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc. Figure 32 As shown, the computer device 10000 includes, but is not limited to, at least the following: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked to each other via a system bus. Wherein:

[0267] The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of a computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for scheduling methods, DAG configuration methods for video processing, DAG configuration methods for document processing, DAG-based document processing methods, or video processing methods based on video DAG. Furthermore, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0268] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0269] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0270] It should be pointed out that, Figure 32 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0271] In this embodiment, the scheduling method stored in memory 10010 can be further divided into one or more program modules and executed by one or more processors (processor 10020 in this embodiment) to complete the embodiment of this application.

[0272] Example 4

[0273] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the scheduling method in the embodiments.

[0274] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the scheduling method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0275] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0276] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A scheduling method for a DAG scheduling server, characterized in that, The method includes: Receive a call request, the call request including request parameters indicating the call triggering method; The target DAG is obtained according to the call request, and the target DAG includes multiple processing units with dependencies. Target scheduling is performed according to the call triggering method and the target DAG to execute the processing unit to be executed among the multiple processing units through the executor; The processing unit to be executed corresponds to standard mode, synchronous mode, or asynchronous mode: The processing unit corresponding to the standard mode includes a synchronous processing part and an asynchronous processing part; The processing unit corresponding to the synchronous mode includes a synchronous processing part, which is first set to the registered state when scheduled and set to the ready state when called; when preparing to execute the synchronous processing part, the processing unit of the synchronous mode is set to the running state, and after execution is completed, it is set to the waiting state. The processing unit corresponding to the asynchronous mode includes an asynchronous processing part, which is first set to the registered state when scheduled, and set to the waiting state when called until it is set to the ready state after receiving a scheduling message. If the processing unit to be executed does not include the synchronous processing part, the processing unit to be executed is directly set to a waiting state to save the step of idling execution of the executor and the database exchange between the DAG scheduling server and the executor due to unit status verification during idling.

2. The scheduling method according to claim 1, characterized in that, The step of executing target scheduling based on the call triggering method and the target DAG includes: The processing unit to be executed is determined based on the target DAG, and the unit state of the processing unit to be executed is set or updated; and If the unit status setting or update of the processing unit to be executed is completed, the executor is notified to execute the processing unit to be executed.

3. The scheduling method according to claim 2, characterized in that, The triggering method is resource addition; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit status of the processing unit to be executed includes: Identify the processing units in the target DAG with an in-degree of 0, and designate these processing units with an in-degree of 0 as the processing units to be executed; and The DAG state server is requested to update the unit status of the processing unit to be executed to the ready state.

4. The scheduling method according to claim 2, characterized in that, The triggering method is resource update; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit state of the processing unit to be executed includes: The processing unit to be executed is determined based on the target DAG; and The system requests the DAG state server to update the unit status of the processing unit to be executed to the ready state, and also requests the DAG state server to update the unit status of each processing unit downstream of the processing unit to the registered state.

5. The scheduling method according to claim 2, characterized in that, The triggering method is resource deletion; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit status of the processing unit to be executed includes: Each processing unit of the target DAG is determined as the processing unit to be executed; and The DAG state server is requested to update the unit status of the processing unit to be executed to the ready state.

6. The scheduling method according to claim 2, characterized in that, The triggering method is a resource callback; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit state of the processing unit to be executed includes: Based on the target DAG and the resource callback, determine the processing unit to be executed; and The DAG state server is requested to update the unit status of the processing unit to be executed to the ready state.

7. The scheduling method according to claim 4 or 6, characterized in that, The method further includes: Obtain the unit status of the processing unit to be executed from the DAG state server to determine whether to invoke it.

8. The scheduling method according to claim 2, characterized in that, The triggering method is completed by the processing unit; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit state of the processing unit to be executed includes: Based on the target DAG and the completed processing units corresponding to the processing unit, the downstream processing units of the completed processing units are determined as the processing units to be executed; and The DAG state server is requested to update the unit status of the processing unit to be executed to either the ready state or the waiting state.

9. The scheduling method according to claim 2, characterized in that, The triggering method is to re-execute the target DAG; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit state of the processing unit to be executed includes: Identify the processing units in the target DAG with an in-degree of 0, and designate these processing units with an in-degree of 0 as the processing units to be executed; and The system requests the DAG state server to update the unit status of the processing unit to be executed to the ready state, and requests the DAG state server to update the other processing units in the target DAG, excluding the processing unit to be executed, to the registered state.

10. The scheduling method according to claim 2, characterized in that, The triggering method is to re-execute the specified processing unit; the step of determining the processing unit to be executed based on the target DAG and setting or updating the unit state of the processing unit to be executed includes: The specified processing unit in the target DAG is determined as the processing unit to be executed; and The system requests the DAG state server to update the unit status of the processing unit to be executed to the ready state, and also requests the DAG state server to update the downstream processing units of the processing unit to be executed in the target DAG to the registered state.

11. A scheduling device for a DAG scheduling server, characterized in that, The device includes: The first acquisition module is used to acquire a call request, wherein the call request includes request parameters indicating the call triggering method; The second acquisition module is used to acquire the target DAG according to the call request, wherein the target DAG includes multiple processing units with dependencies. An execution module is used to perform target scheduling according to the call triggering method and the target DAG, so as to execute the processing unit to be executed among the multiple processing units through the executor; The processing unit to be executed corresponds to standard mode, synchronous mode, or asynchronous mode: The processing unit corresponding to the standard mode includes a synchronous processing part and an asynchronous processing part; The processing unit corresponding to the synchronous mode includes a synchronous processing part, which is first set to the registered state when scheduled and set to the ready state when called; when preparing to execute the synchronous processing part, the processing unit of the synchronous mode is set to the running state, and after execution is completed, it is set to the waiting state. The processing unit corresponding to the asynchronous mode includes an asynchronous processing part, which is first set to the registered state when scheduled, and set to the waiting state when called until it is set to the ready state after receiving a scheduling message. If the processing unit to be executed does not include the synchronous processing part, the processing unit to be executed is directly set to a waiting state to save the step of idling execution of the executor and the database exchange between the DAG scheduling server and the executor due to unit status verification during idling.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the scheduling method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the scheduling method according to any one of claims 1 to 10.

14. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the scheduling method according to any one of claims 1 to 10.

Images