Workflow scheduling method and device, computer readable storage medium

Abstract

The present disclosure relates to a workflow scheduling method and device, and a computer readable storage medium. The workflow scheduling method comprises: defining a workflow according to user input, wherein the workflow is a process arranged for a plurality of tasks, the plurality of tasks are tasks included in a model training whole process, and the workflow comprises a plurality of tasks in data labeling and model evaluation in the model training whole process; and scheduling execution of the workflow, wherein for different tasks, pre-configured program data in shared storage is run according to a configured type to execute the tasks. In the labeling and effect evaluation process in the model training process, the present disclosure can automatically link the preceding and subsequent tasks, thereby reducing the communication cost of algorithm personnel and labeling personnel.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of workflow scheduling, and in particular, to a workflow scheduling method and device, and a computer readable storage medium. BACKGROUND

[0002] In the related art deep learning model iterative updating, for some businesses, the distribution of data changes over time, such as new words and sentence patterns being created every day, so the model needs to be iteratively updated to ensure the online effect.

[0003] For such a model, the typical steps of one iteration include: 1, original data collection & preprocessing; 2, data labeling, obtaining labeled data; 3, model training, obtaining the final model; 4, model evaluation, comparing the differences in the effects of new and old models on the new data set. Among them, the second and fourth steps need human participation, generally algorithm personnel and labeling personnel communicate, and after the labeling personnel complete, the algorithm personnel are notified to continue the next step. SUMMARY

[0004] The inventors have found that in the related art model training process, the labeling and effect evaluation parts that depend on human cannot automatically connect the previous and next tasks.

[0005] In view of at least one of the above technical problems, the present disclosure provides a workflow scheduling method and device, and a computer readable storage medium, which can automatically connect the previous and next tasks in the labeling and effect evaluation process in the model training process.

[0006] According to one aspect of the present disclosure, a workflow scheduling method is provided, comprising:

[0007] defining a workflow according to the input of a user, wherein the workflow is a process of arranging a plurality of tasks, the plurality of tasks are tasks included in a model training whole process, and the workflow includes a plurality of tasks in data labeling and model evaluation in the model training whole process;

[0008] scheduling to execute the workflow, wherein for different tasks, according to the type configured, a pre-configured program data in a shared storage is run to execute the task.

[0009] In some embodiments of the present disclosure, the workflow is a directed acyclic graph.

[0010] In some embodiments of the present disclosure, the workflow scheduling method further comprises:

[0011] preparing program data of specific task code;

[0012] The program data of the specific task code is stored in the corresponding specific path of the shared storage.

[0013] In some embodiments of this disclosure, the specific task code includes sending a notification message and accessing any item in a data statistics table.

[0014] In some embodiments of this disclosure, the workflow scheduling method further includes:

[0015] The results of the workflow are saved in shared storage;

[0016] Based on the user's request, the workflow execution results are sent to the user.

[0017] In some embodiments of this disclosure, defining the workflow based on user input includes:

[0018] Define the workflow based on user input;

[0019] Determine the task type based on runtime parameters and define each task in the workflow;

[0020] Arrange multiple tasks in the workflow according to at least one of the relationships of parallelism and serialism.

[0021] In some embodiments of this disclosure, each task in the defined workflow includes:

[0022] During task definition, module resource parameters are added to the task, and the resources corresponding to the template are configured on the server. During task execution, the parameters corresponding to the module resource parameters are used.

[0023] In some embodiments of this disclosure, the workflow scheduling method further includes:

[0024] If the predetermined retry conditions are met, the workflow is rescheduled to be executed starting from the specific task.

[0025] In some embodiments of this disclosure, the predetermined retry condition includes at least one of the following conditions, wherein:

[0026] A task is executed successfully, but a logical error is found that requires code modification.

[0027] A task requires insufficient resources and the resource size needs to be reset.

[0028] A task encountered a memory overflow and the resource size needs to be reset.

[0029] In some embodiments of this disclosure, rescheduling the execution of the workflow from a specific task includes:

[0030] starting from a specific task, traversing the workflow of the directed acyclic graph, taking the traversed nodes as nodes that need to be re-run;

[0031] taking the union of all the traversed nodes;

[0032] in the task scheduling process, skipping the nodes not in the union.

[0033] In some embodiments of the present disclosure, the workflow scheduling method further comprises:

[0034] indicating that the interactive task platform opens a first interface and a second interface, wherein the first interface is used to publish tasks, and the second interface is used to query whether the task is completed;

[0035] making data required by the interactive task platform;

[0036] sending the data required by the interactive task platform to the interactive platform;

[0037] receiving an identifier of the interactive task returned by the first interface, and sending the identifier of the interactive task to the corresponding interactive personnel;

[0038] cyclically querying the interactive task platform through the second interface to query whether the interactive task is completed.

[0039] In some embodiments of the present disclosure, the interactive task is a labeling task.

[0040] In some embodiments of the present disclosure, the cyclically querying the interactive task platform through the second interface to query whether the interactive task is completed comprises:

[0041] cyclically querying the interactive task platform through the second interface at a predetermined query interval to query whether the interactive task is completed.

[0042] In some embodiments of the present disclosure, the workflow scheduling method further comprises:

[0043] if it is determined that the interactive task is not completed by cyclically querying the interactive task platform through the second interface within a predetermined timeout period, sending a reminder message to the interactive personnel.

[0044] In some embodiments of the present disclosure, the workflow scheduling method further comprises:

[0045] in the case of task success, task failure, task completion, start of manual interaction or end of manual interaction, sending a notification message to the relevant personnel.

[0046] According to another aspect of the present disclosure, a workflow scheduling device is provided, comprising:

[0047] A workflow definition module configured to define a workflow according to an input of a user, wherein the workflow is a process of orchestrating a plurality of tasks, and the workflow includes a plurality of tasks in data labeling and model evaluation in a model training whole process.

[0048] A workflow scheduling module configured to schedule execution of the workflow, wherein for different tasks, a pre-configured program data in a shared storage is run according to a configured type to execute the task.

[0049] According to another aspect of the present disclosure, a workflow scheduling device is provided, comprising:

[0050] a memory configured to store instructions; and

[0051] a processor configured to execute the instructions to implement the workflow scheduling method according to any one of the above embodiments.

[0052] According to another aspect of the present disclosure, a computer readable storage medium is provided, wherein the computer readable storage medium stores computer instructions, and the instructions are executed by a processor to implement the workflow scheduling method according to any one of the above embodiments.

[0053] The present disclosure can automatically link the tasks before and after in the process of labeling and effect evaluation in the model training process, and reduce the communication cost of algorithm personnel and labeling personnel. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0055] Figure 1 A schematic diagram of some embodiments of the workflow scheduling method of the present disclosure.

[0056] Figure 2 A schematic diagram of some other embodiments of the workflow scheduling method of the present disclosure.

[0057] Figure 3 A schematic diagram of some other embodiments of the workflow scheduling method of the present disclosure.

[0058] Figure 4 A schematic diagram of some embodiments of the workflow scheduling device of the present disclosure.

[0059] Figure 5 A schematic diagram of some other embodiments of the workflow scheduling device of the present disclosure. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present disclosure will be described clearly and completely in combination with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting on the present disclosure and its applications or uses. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present disclosure.

[0061] Unless specifically stated otherwise, the relative arrangements of the components and steps illustrated in these embodiments do not limit the scope of the present disclosure.

[0062] It should be understood that the sizes of the various portions shown in the drawings are not necessarily drawn to scale for ease of description.

[0063] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail in order to avoid obscuring the present disclosure.

[0064] In all the examples shown and discussed herein, any specific values should be interpreted as merely illustrative and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0065] It should be noted that like reference numerals and letters refer to like items in the following drawings, and thus, once an item is defined in one drawing, it need not be discussed further in subsequent drawings.

[0066] In a related technology, a workflow scheduling framework of DolphinScheduler is adopted. DolphinScheduler is an open-source distributed workflow scheduling system designed to help users automate the scheduling, execution, and monitoring of complex data processing and computing tasks. It supports visual workflow design, task orchestration, and scheduling, with the characteristics of easy-to-use and scalability. DolphinScheduler supports multiple task types, including data extraction, data processing, machine learning, etc., and provides rich monitoring and alarm functions so that users can track the execution status and performance of tasks in real time.

[0067] Another related technology uses Kubeflow Pipeline. Kubeflow is an open-source platform designed specifically for ML (Machine Learning) workflows and deployment, which provides a powerful set of tools and components on Kubernetes. Kubeflow Pipeline is a component for scheduling workflows.

[0068] DolphinScheduler focuses more on distributed task scheduling and monitoring, suitable for various computing environments; while Kubeflow Pipeline focuses on the orchestration and execution of containerized tasks in Kubernetes clusters, suitable for cloud-native application development and deployment.

[0069] The inventors have also found that both of the above workflow scheduling systems provide Python (a computer programming language) SDK (Software Development Kit), but are not convenient to use and require a lot of time to learn, which will distract the attention of algorithm developers.

[0070] The related technology DolphinScheduler is relatively difficult to deploy and operate, and the running nodes need to install dependent environments, and the installation of deep learning module environments is also complex, so it is not suitable for deep learning scenarios.

[0071] Using the related technology Kubeflow Pipeline also requires understanding some kubernetes concepts, and when the number of tasks increases, the configuration becomes complex.

[0072] The related technology requires modifying the dependencies of downstream tasks when adding or deleting a task when building a DAG (Directed Acyclic Graph), which makes it easy to make mistakes when modifying the workflow. Although it is a platform specifically designed for machine learning, it still cannot easily handle processes that require human interaction.

[0073] In view of at least one of the above technical problems, the present disclosure provides a workflow scheduling method and device, and a computer readable storage medium. The present disclosure will be described below through specific embodiments.

[0074] Figure 1 A schematic diagram of some embodiments of the workflow scheduling method of the present disclosure. Preferably, the present embodiment can be executed by the workflow scheduling device of the present disclosure. The method can at least include at least one of steps 11 to 12, wherein:

[0075] Step 11, according to the input of the user, the definition of the workflow is performed, wherein the workflow is a process in which a plurality of tasks included in the model training process are arranged.

[0076] In some embodiments of the present disclosure, the plurality of tasks can include a plurality of tasks in data labeling and model evaluation in the model training process.

[0077] The present disclosure can automatically connect the tasks before and after the labeling and effect evaluation in the model training process, thereby reducing the communication cost of algorithm personnel and labeling personnel.

[0078] In some embodiments of the present disclosure, the workflow can be a directed acyclic graph.

[0079] In some embodiments of the present disclosure, the workflow can be a process in which a plurality of tasks are arranged, and the process can be described as a directed acyclic graph. Figure 2 A schematic diagram of the directed acyclic graph in some embodiments of the present disclosure is shown in FIG. 1. Figure 2 As shown in the figure, each node in the figure represents a task.

[0080] In some embodiments of the present disclosure, the workflow can be a cloud-native workflow.

[0081] Step 12, scheduling and executing the workflow, wherein for different tasks, according to the type of configuration, the pre-configured program data in the shared storage is run to execute the task.

[0082] In some embodiments of the present disclosure, step 12 can include: the workflow scheduling device schedules the tasks according to the DAG to ensure the running order between the tasks.

[0083] In some embodiments of the present disclosure, with the rise of the cloud-native ecosystem, the workflow technology is also developing towards the cloud-native direction.

[0084] Figure 2 A schematic diagram of another embodiment of the workflow scheduling method of the present disclosure is shown in FIG. 2. Preferably, the present embodiment can be executed by the workflow scheduling device of the present disclosure. The method can include at least one of steps 10 to 16, wherein:

[0085] Step 10, pre-preparing program data of a specific task code; and storing the program data of the specific task code on a corresponding specific path of the shared storage.

[0086] In some embodiments of the present disclosure, the specific task code includes any one of sending a notification message and accessing a data statistics table (accessing Hive), wherein Hive is a data statistics tool opened by Facebook for solving massive structured logs, and Hive is a data warehouse tool based on Hadoop, which can map structured data files into a table and provide a SQL-like query function.

[0087] In some embodiments of the present disclosure, step 10 can include presetting task types such as artificial labeling / Hive reading / running Gitlab (a code and document management tool) code, and components required by the whole process of model training can be ready for use out of the box.

[0088] In some embodiments of the present disclosure, step 10 can include that shared storage can be used on kubernetes, so that all tasks can access data files of the shared storage during runtime. The present disclosure places common task codes (such as accessing Hive and sending a notification message) on a specific path of the shared storage, and the task entry code only needs to load the specific path to sys.path, so that these tasks can be used, thereby eliminating the need to preinstall these task codes in the image.

[0089] The present disclosure utilizes shared storage to perform predefined task type codes on a kubernetes cluster.

[0090] Step 11, according to the input of a user, defining a workflow, wherein the workflow is a process of arranging a plurality of tasks included in the whole process of model training.

[0091] In some embodiments of the present disclosure, step 11 can include defining a task in the SDK by constructing an instance of a class.

[0092] In some embodiments of the present disclosure, step 11 can include at least one of steps 111 to 113, wherein:

[0093] Step 111, according to the input of a user, defining a workflow.

[0094] In some embodiments of the present disclosure, step 111 can include that a user uses the SDK to write a python file to define a workflow.

[0095] The present disclosure designs a mechanism for defining a workflow DAG through a python operator.

[0096] In some embodiments of the present disclosure, step 111 can include defining a workflow name and an owner of the workflow according to the input of the user.

[0097] The following is illustrated by way of example 1. Example 1 is a schematic for defining a workflow in some embodiments of the present disclosure.

[0098] Example 1

[0099] owner = 'owner_name'

[0100] flow = Workflow(name = 'demo-flow', owner = owner)

[0101] Step 112, determine the type of task according to the runtime parameter, and define each task in the workflow.

[0102] In some embodiments of the present disclosure, step 21 can include establishing a separate directory for each workflow.

[0103] The following is illustrated by way of example 2. Example 2 is a schematic for defining a task in a workflow in some embodiments of the present disclosure.

[0104] Example 2

[0105]

[0106]

[0107] In some embodiments of the present disclosure, the step of defining each task in the workflow can include adding a module resource parameter in the task during task definition, and configuring the corresponding resource of the template on the server, wherein the parameter corresponding to the module resource parameter is used during task execution.

[0108] In some embodiments of the present disclosure, the workflow scheduling method can include shielding the algorithm personnel from understanding kubernetes.

[0109] In some embodiments of the present disclosure, the workflow scheduling method can include defining some commonly used resource templates on the server, and adding a resource_template (template corresponding resource) parameter in the Task during task definition, indicating the use of the corresponding template. Thus, the corresponding resources of the template (such as the selector / tolerations (selector / tolerations, tolerance) parameters of kubernetes) can be configured on the server, thereby shielding these concepts, as shown in example 3, indicating that the applicant owns a node with two V100-GPU cards.

[0110] Embodiment 3

[0111]

[0112] The above embodiments of the present disclosure reduce the understanding of the user for the kubernetes concept, enable the user to use the system to build the entire training process more quickly and simply, and for the common task type, the algorithm personnel only need to specify the necessary parameters to use.

[0113] In step 113, a plurality of tasks in the workflow are arranged according to at least one of parallel and serial relationship.

[0114] In some embodiments of the present disclosure, step 113 can include: a set of DAG definition methods is designed, and a python operator (>> or |) is overloaded to define the serial and parallel relationship between tasks.

[0115] For example: task1>>(task2|task3|task4)>>task5, which means that after task1 is executed, task2 / task3 / task4 are executed in parallel, and after these tasks are executed, task5 is executed.

[0116] The following is described by Embodiment 4. Embodiment 4 is a serial relationship in some embodiments of the present disclosure.

[0117] Embodiment 4

[0118] flow.arrange(echo>>ding_send) # first execute echo, and then execute ding_send

[0119] flow.start()

[0120] The above embodiments of the present disclosure use the operator overload to construct the DAG, which can very simply define the DAG that is not too complex, and can meet more than 90% of the demand scenarios.

[0121] In some embodiments of the present disclosure, step 113 can include: a definition mode is provided to meet the same arrangement function: Seq(task1,Parallel(task2,task3,task4),task5), which is convenient for arranging a large number of tasks.

[0122] In step 12, the workflow is scheduled and executed, wherein for different tasks, according to the configured type, the pre-configured program data in the shared storage is run to execute the task.

[0123] Step 13, in the case of meeting a predetermined retry condition, rescheduling execution of the workflow from a specific task.

[0124] In some embodiments of the present disclosure, step 13 can include dynamically modifying the workflow and then retrying execution from certain tasks.

[0125] In some embodiments of the present disclosure, the predetermined retry condition can include at least one of the following conditions:

[0126] Condition one, a task is successfully executed, and it is found that there is a logical error in the task and the code needs to be modified.

[0127] Condition two, the resources required by a task are insufficient, and the resource size needs to be reset.

[0128] Condition three, a task has an out-of-memory (OOM) error, and the resource size needs to be reset.

[0129] During the execution of the workflow, there are some cases that need to be retried. Although Argo provides a mechanism to retry from a failed task, it does not provide a method to retry from a successfully executed task, nor does it provide a method to restart after modifying the resource required by a node. In the SDK of the present disclosure, start_from can be used to specify which tasks to start from, which is very useful for debugging the workflow. The above embodiments of the present disclosure are applicable to the following situations: when a task is successfully executed, but it is found that there is a logical error and the code needs to be modified for re-execution; when it is found that the resources required by a task are insufficient, and OOM and other phenomena occur, the resource size needs to be reset for retry.

[0130] The following is illustrated by embodiment 5. Embodiment 5 is an example of dynamically modifying the workflow and then retrying execution from certain tasks in some embodiments of the present disclosure.

[0131] Embodiment 5

[0132] flow.start_from(ding_send) # start execution from ding_send

[0133] In some embodiments of the present disclosure, in step 13, the step of rescheduling execution of the workflow from a specific task can include at least one of steps 131 to 133, wherein:

[0134] Step 131, starting from a specific task, traversing the directed acyclic graph of the workflow, and taking the nodes traversed as nodes that need to be re-run.

[0135] Step 132, taking the union of all the nodes traversed.

[0136] Step 133, in the task scheduling process, skipping the nodes not in the union.

[0137] The above embodiments of the present disclosure design a mechanism for starting execution from a specified task restart and skipping the preceding tasks.

[0138] The above embodiments of the present disclosure adopt a mechanism for retrying execution from certain nodes after dynamically modifying the workflow.

[0139] Step 14, opening the artificial interactive task.

[0140] In some embodiments of the present disclosure, step 14 can include at least one of steps 141-145, wherein:

[0141] Step 141, instructing the interactive task platform to open a first interface and a second interface, wherein the first interface is used to publish tasks, and the second interface is used to query whether the task is completed.

[0142] In some embodiments of the present disclosure, the first interface can be an HTTP interface.

[0143] In some embodiments of the present disclosure, the interactive task can be an interactive task such as a labeling task; and the interactive task platform can be a labeling platform.

[0144] Step 142, making data required by the interactive task platform.

[0145] In some embodiments of the present disclosure, step 142 can be completed by running custom code.

[0146] Step 143, sending the data required by the interactive task platform to the interactive platform.

[0147] Step 144, receiving an identifier of the interactive task returned by the first interface, and sending the identifier of the interactive task to the corresponding interactive personnel.

[0148] In some embodiments of the present disclosure, the interactive task can be an interactive task such as a labeling task; the interactive task platform can be a labeling platform; and the interactive personnel can be a labeling personnel.

[0149] Step 145, cyclically querying the interactive task platform through the second interface to query whether the interactive task is completed.

[0150] In some embodiments of the present disclosure, step 145 can include cyclically querying the interactive task platform through the second interface at a predetermined query interval to query whether the interactive task is completed.

[0151] In some embodiments of the present disclosure, the workflow scheduling method can further include: if the interactive task is not completed within a predetermined timeout period, the reminder message is sent to the interactive personnel by cyclically querying the interactive task platform through the second interface.

[0152] In some embodiments of the present disclosure, step 145 can include: cyclically querying the labeling platform to determine whether the task is completed, and a timeout period and a query interval can be set.

[0153] In the above embodiments of the present disclosure, in addition to the labeling platform, all tasks requiring human interaction only need to abstract the interface of HTTP submission and query, and can be implemented through task scheduling, thereby avoiding the notification and delivery of cooperating personnel.

[0154] Step 15, in the case of task success, task failure, task completion, human interaction start or human interaction end, a notification message is sent to the relevant personnel.

[0155] The above embodiments of the present disclosure integrate the part requiring human interaction into the workflow through the submission / waiting interface and send a notification. Thus, the communication cost is reduced, the overall efficiency is improved, and the user experience is improved.

[0156] The above embodiments of the present disclosure add a convenient message notification mechanism, and send a notification message (for example, through Dingding, Feishu, etc.) to the relevant personnel when the task is successful or failed, or the human interaction starts or ends.

[0157] Step 16, save the running result of the workflow in a shared storage; and according to the request of the user, send the saved running result of the workflow to the user.

[0158] The method of the above embodiments of the present disclosure can be used for the concatenation of Hive table query, model training, business index statistics and other processes of multiple business lines. The above embodiments of the present disclosure can save 100 person-days / month of manpower of internal algorithm personnel. In the above embodiments of the present disclosure, five business models are automatically updated, and the communication cost between algorithm personnel and labeling personnel is saved.

[0159] Figure 3 A schematic diagram of still another embodiment of the workflow scheduling method of the present disclosure. Preferably, the present embodiment can be executed by the workflow scheduling device of the present disclosure. The method can include at least one of steps 1 to 7, wherein:

[0160] Step 1, receive the input of the user, and generate a python file according to the input of the user using the SDK language to define the workflow.

[0161] Step 2, run the python file, and send the DAG to the workflow server.

[0162] Step 3, the workflow server parses the DAG, converts it into a workflow of Argo Workflow, and sends it to the argo-server.

[0163] Step 4, the argo-server schedules this DAG in the kubernetes cluster.

[0164] Step 5, different tasks run different codes according to the configured types, and read and write data in the shared storage

[0165] Step 6, the user is notified after the task is completed.

[0166] Step 7, the user can download the running results from the shared storage.

[0167] The above embodiments of the present disclosure develop a general cloud-native workflow engine architecture based on Argo Workflow. This architecture customizes some task types for some processes in algorithm model training, and provides a Python SDK to make it more convenient for users to use, wherein Argo Workflow is an open source container-native workflow engine that can orchestrate parallel jobs on Kubernetes.

[0168] Figure 4 A schematic diagram of some embodiments of the workflow scheduling apparatus of the present disclosure is shown. As shown in the figure, the workflow scheduling apparatus of the present disclosure can include a workflow definition module 41 and a workflow scheduling module 42, wherein: Figure 4

[0169] The workflow definition module 41 is configured to define a workflow according to the input of a user, wherein the workflow is a process of orchestrating a plurality of tasks, the plurality of tasks are tasks included in a model training whole process, and the workflow includes a plurality of tasks in data labeling and model evaluation in the model training whole process.

[0170] In some embodiments of the present disclosure, the workflow is a directed acyclic graph.

[0171] In some embodiments of the present disclosure, the workflow definition module 41 is configured to define a workflow according to the input of a user, determine a task type according to a runtime parameter, define each task in the workflow, and orchestrate a plurality of tasks in the workflow in at least one of a parallel and a serial relationship.

[0172] ​In some embodiments of the present disclosure, the workflow definition module 41, in defining each task in the workflow, is configured to add a module resource parameter in the task definition process, and configure the corresponding resource of the template on the server side, wherein the module resource parameter is used to configure the parameter corresponding to the module resource parameter in the task execution process.

[0173] The workflow scheduling module 42 is configured to schedule the execution of the workflow, wherein, for different tasks according to the configured type, the pre-configured program data in the shared storage is run to execute the task.

[0174] In some embodiments of the present disclosure, as shown in Figure 4 The workflow scheduling device of the present disclosure can include a data pre-configuration module 43, wherein:

[0175] The data pre-configuration module 43 is configured to pre-configure the program data of the specific task code, and store the program data of the specific task code on the corresponding specific path of the shared storage.

[0176] In some embodiments of the present disclosure, the specific task code includes any one of sending a notification message and accessing a data statistics table.

[0177] In some embodiments of the present disclosure, as shown in Figure 4 The workflow scheduling device of the present disclosure can include a running result processing module 44, wherein:

[0178] The running result processing module 44 is configured to save the running result of the workflow in the shared storage, and send the running result of the workflow to the user according to the request of the user.

[0179] In some embodiments of the present disclosure, as shown in Figure 4 The workflow scheduling device of the present disclosure can include a rescheduling module 45, wherein:

[0180] The rescheduling module 45 is configured to reschedule the execution of the workflow from a specific task when a predetermined retry condition is met.

[0181] In some embodiments of the present disclosure, the predetermined retry condition can include at least one of the following conditions: one task is executed successfully, and it is found that the task has a logical error and needs to be modified; the resource required by one task is insufficient and needs to be reconfigured; one task has a memory overflow and needs to be reconfigured.

[0182] In some embodiments of the present disclosure, the rescheduling module 45 is configured to, in the case that a predetermined retry condition is met, start from a specific task, traverse the workflow of the directed acyclic graph, take the traversed nodes as nodes that need to be re-run, take the union of all the traversed nodes, and skip the nodes that are not in the union in the task scheduling process.

[0183] In some embodiments of the present disclosure, as shown in Figure 4 The workflow scheduling apparatus of the present disclosure can include an interaction module 46, wherein:

[0184] The interaction module 46 is configured to instruct an interaction task platform to open a first interface and a second interface, wherein the first interface is used for publishing tasks, and the second interface is used for querying whether a task is completed; make data required by the interaction task platform; send the data required by the interaction task platform to an interaction platform; receive an identifier of an interaction task returned by the first interface, and send the identifier of the interaction task to a corresponding interaction personnel; and cyclically query the interaction task platform through the second interface to query whether the interaction task is completed.

[0185] In some embodiments of the present disclosure, the interaction task is a labeling task.

[0186] In some embodiments of the present disclosure, in the case that the interaction module 46 cyclically queries the interaction task platform through the second interface to query whether the interaction task is completed, the interaction module 46 is configured to cyclically query the interaction task platform through the second interface to query whether the interaction task is completed at a predetermined query interval.

[0187] In some embodiments of the present disclosure, the interaction module 46 can also be configured to, if it is determined through the second interface that the interaction task is not completed within a predetermined timeout period, send a reminder message to the interaction personnel.

[0188] In some embodiments of the present disclosure, as shown in Figure 4 The workflow scheduling apparatus of the present disclosure can include a notification module 47, wherein:

[0189] The notification module 47 is configured to send a notification message to relevant personnel in the case that a task is successful, a task fails, a task is completed, human interaction starts, or human interaction ends.

[0190] Figure 3 Another schematic diagram of another embodiment of the workflow scheduling apparatus of the present disclosure is given. As shown in Figure 3 The workflow scheduling apparatus of the present disclosure can include a workflow definition module 31, a workflow service end 32, a server 33, a function integration module 34, a shared storage 35, a memory 36, and an image harbor 37, wherein:

[0191] The workflow definition module 31 is configured to receive the input of the user, generate a python file according to the input of the user using the SDK language, and define the workflow; run the python file, and send a directed acyclic graph (DAG) to the workflow server 32.

[0192] In some embodiments of the present disclosure, the workflow definition module 31 can be implemented as a Python SDK.

[0193] The workflow server 32 is configured to parse the DAG, convert it into the workflow of Argo Workflow, and send it to the server 33.

[0194] In some embodiments of the present disclosure, the workflow server 32 can also be configured to store the workflow of Argo Workflow into the memory 36 in a snapshot manner.

[0195] The server 33 is configured to schedule the DAG in a kubernetes cluster.

[0196] In some embodiments of the present disclosure, the workflow server 32 can be implemented as an Argo server.

[0197] In some embodiments of the present disclosure, the workflow server 32 can also be configured to store the running record of the DAG into the memory 36.

[0198] The function integration module 34 is configured to integrate different tasks.

[0199] In some embodiments of the present disclosure, as shown in FIG. 2, the function integration module 34 is configured to integrate a message notification task, a Hive-SQL (Structured Query Language) task, a labeling task, a Git (version control system) code running task, a Spark (general-purpose distributed data processing engine) task, an Http (Transfer Protocol) task, and the like. Figure 3

[0200] The shared storage 35 is configured to pre-store the program data of specific task codes and the log data of the DAG scheduling.

[0201] In some embodiments of the present disclosure, the pre-stored program data of specific task codes can be a preset Python package.

[0202] ​In some embodiments of this disclosure, log data is stored in a workspace path, such as: workplace: / root / data / argo / flowname / .

[0203] In some embodiments of this disclosure, server 33 may also be configured to run different code according to the type of different tasks configured in functional integration module 34 when scheduling the DAG, and read and write data in shared storage 34; and notify the user when the task is completed.

[0204] In some embodiments of this disclosure, the shared storage 35 can also be configured to send the workflow execution results to the user upon request. The user can then download the execution results from the shared storage.

[0205] Figure 5 This is a schematic diagram illustrating the structure of some embodiments of the workflow scheduling device disclosed herein. For example... Figure 5 As shown, the item positioning device includes a memory 51 and a processor 52.

[0206] Memory 51 is used to store instructions, and processor 52 is coupled to memory 51. Processor 52 is configured to execute instructions stored in memory to implement any of the above embodiments (e.g., Figures 1-3 The workflow scheduling method described in any embodiment.

[0207] like Figure 5 As shown, the workflow scheduling device also includes a communication interface 53 for information exchange with other devices. Simultaneously, the item positioning device also includes a bus 54, through which the processor 52, communication interface 53, and memory 51 communicate with each other.

[0208] The memory 51 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. The memory 51 may also be a memory array. The memory 51 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0209] Furthermore, processor 52 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0210] According to another aspect of this disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement any of the embodiments described above (e.g., Figures 1 to 3 The workflow scheduling method described in any embodiment.

[0211] In some embodiments of the disclosure, the computer readable storage medium can be a non-transitory computer readable storage medium.

[0212] Those skilled in the art will appreciate that embodiments of the disclosure can be supplied as a method, device, or computer program product. Accordingly, the disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the disclosure can take the form of a computer program product on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer readable program code thereon.

[0213] The disclosure is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the disclosure. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as a combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the flow Figure 1 The flow or flows and / or blocks in the flowchart and / or block diagram Figure 1 The device that implements the functions specified in the flow or flows and / or blocks in the flowchart and / or block diagram.

[0214] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device that implements the flow Figure 1 The flow or flows and / or blocks in the flowchart and / or block diagram Figure 1 The device that implements the functions specified in the flow or flows and / or blocks in the flowchart and / or block diagram.

[0215] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the flow Figure 1 The flow or flows and / or blocks in the flowchart and / or block diagram Figure 1 The device that implements the functions specified in the flow or flows and / or blocks in the flowchart and / or block diagram.

[0216] The workflow definition module and the workflow scheduling module, the data pre-establishing module, the running result processing module, the rescheduling module, the interaction module, the workflow definition module, the workflow server, the server, the function integration module and the shared storage described above can be implemented as a general processor, a programmable logic controller (PLC), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component or any appropriate combination thereof for performing the functions described in the present application.

[0217] So far, the present disclosure has been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein according to the above description.

[0218] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing related hardware to complete, and the program can be stored in a non-transitory computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0219] The description of the present disclosure is given for the purpose of illustration and description, and is not exhaustive or limiting to the present disclosure. Many modifications and variations will be apparent to those skilled in the art. The embodiments are chosen and described in order to better illustrate the principles and practical application of the present disclosure, and to enable those skilled in the art to understand the present disclosure in order to design various embodiments with various modifications for specific purposes.

Claims

1. A workflow scheduling method, comprising: defining a workflow according to a user input, wherein the workflow is a process of orchestrating a plurality of tasks included in a model training whole process, and the workflow includes a plurality of tasks in data labeling and model evaluation in the model training whole process; scheduling execution of the workflow, wherein for different tasks, a pre-configured program data in a shared storage is run according to a configured type to execute the task; instructing an interactive task platform to open a first interface and a second interface, wherein the first interface is used for publishing a task, and the second interface is used for querying whether the task is completed; making data required by the interactive task platform; sending the data required by the interactive task platform to the interactive platform; receiving an identifier of an interactive task returned by the first interface, and sending the identifier of the interactive task to a corresponding interactive personnel, wherein the interactive task is a labeling task; cyclically querying the interactive task platform through the second interface to query whether the interactive task is completed; re-scheduling execution of the workflow from a specific task in a case where a predetermined retry condition is met, wherein the predetermined retry condition includes at least one of the following conditions: one task is executed successfully, and it is found that the task has a logical error and needs to be modified; one task lacks resources and needs to be re-sized; one task has a memory overflow and needs to be re-sized.

2. The workflow scheduling method of claim 1, wherein, The workflow is a directed acyclic graph. 3.The workflow scheduling method of claim 1 or 2, further comprising: preparing program data of a specific task code; storing the program data of the specific task code on a corresponding specific path of the shared storage.

4. The workflow scheduling method of claim 3, wherein, The specific task code includes any one of sending a notification message and accessing a data statistics table.

5. The workflow scheduling method of claim 3, wherein, Further comprising: saving a running result of the workflow in the shared storage; sending the running result of the workflow to a user according to a request of the user.

6. The workflow scheduling method of claim 1 or 2, wherein, The defining the workflow according to the user input comprises: defining the workflow according to the user input; determining a task type according to a runtime parameter to define each task in the workflow; orchestrating the plurality of tasks in the workflow in at least one of parallel and serial relationships.

7. The workflow scheduling method of claim 6, wherein, The defining each task in the workflow comprises: adding a module resource parameter in the task during a task definition process, and configuring a template corresponding resource on a server, wherein a parameter corresponding to the module resource parameter is used during a task execution process.

8. The workflow scheduling method of claim 1 or 2, wherein, The re-scheduling execution of the workflow from the specific task comprises: traversing the workflow of the directed acyclic graph from the specific task, and taking a node traversed as a node needing to be re-run; taking a union set of all the nodes traversed; skipping a node not in the union set during a task scheduling process. 9.The workflow scheduling method of claim 1 or 2, wherein The cyclically querying the interactive task platform through the second interface to query whether the interactive task is completed comprises: cyclically querying the interactive task platform through the second interface to query whether the interactive task is completed at a predetermined query interval.

10. The workflow scheduling method of claim 1 or 2, further comprising: if the interactive task is determined to be not completed by cyclically querying the interactive task platform through the second interface within a predetermined timeout period, sending a reminder message to the interactive personnel.

11. The workflow scheduling method of claim 1 or 2, further comprising: sending a notification message to the relevant personnel in the case of task success, task failure, task completion, start of human interaction or end of human interaction.

12. A workflow scheduling apparatus, comprising: a workflow definition module configured to define a workflow according to an input of a user, wherein the workflow is a process of a plurality of tasks included in a model training whole process, and the workflow includes a plurality of tasks in data labeling and model evaluation in the model training whole process; a workflow scheduling module configured to schedule execution of the workflow, wherein for different tasks, a pre-configured program data in a shared storage is run according to a configured type to execute the task; an interaction module configured to instruct an interactive task platform to open a first interface and a second interface, wherein the first interface is used to publish a task, and the second interface is used to query whether the task is completed; to make data required by the interactive task platform; to send the data required by the interactive task platform to the interactive platform; to receive an identifier of the interactive task returned by the first interface and send the identifier of the interactive task to corresponding interactive personnel, the interactive task being a labeling task; and to cyclically query the interactive task platform through the second interface to query whether the interactive task is completed; a rescheduling module configured to reschedule execution of the workflow from a specific task in the case that a predetermined retry condition is met, wherein the predetermined retry condition includes at least one of the following conditions: one task is executed successfully, and it is found that the task has a logical error and needs to be modified; one task lacks resources and needs to be reconfigured; and one task has a memory overflow and needs to be reconfigured.

13. A workflow scheduling apparatus, comprising: a memory configured to store instructions; and a processor configured to execute the instructions to implement the workflow scheduling method of any one of claims 1-11.

14. A computer readable storage medium, wherein, The computer readable storage medium stores computer instructions, and the instructions are executed by the processor to implement the workflow scheduling method of any one of claims 1-11.

Images