Task scheduling method and device, computer-readable storage medium, and electronic device

Abstract

The present disclosure relates to a task scheduling method and apparatus, a computer-readable storage medium, and an electronic device, and relates to the field of task processing technology. The method comprises: obtaining a blood relationship map of a data processing task, and adjusting the blood relationship map to obtain a one-way chain map of the data processing task; obtaining first resource allocation information of a task node where a data resource required for processing the data processing task is located in the one-way chain map, and a first running time required to process a historical task associated with the data processing task; predicting a second running time required to process the data processing task based on the first resource allocation information and the first running time; and adjusting the running order of the data processing task in the target task list based on the second running time. The present disclosure improves task processing efficiency.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of task processing technology, and in particular to a task scheduling method, a task scheduling device, a computer-readable storage medium, and an electronic device. Background Art

[0002] With the continuous development of computer software and hardware technology, the application areas of parallel distributed computing and large-scale work clusters are also constantly expanding; therefore, how to formulate a scheduling strategy that can balance the load of each node to maximize the utilization of system resources has attracted the attention of more and more relevant practitioners.

[0003] In the existing scheduling strategy generation method, this can be achieved through a round-robin algorithm. Specifically, the round-robin algorithm can linearly rotate among a series of nodes, and the balancer sends the new request to the next node in the scheduling task list, and continuously schedules tasks.

[0004] However, the above method cannot predict the running time required for each data processing task in advance, and therefore cannot dynamically adjust the scheduling task list according to the running time of each task, which will lead to the inability to give priority to data tasks with shorter running times, resulting in low task processing efficiency.

[0005] Therefore, a new task scheduling method needs to be provided.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0007] The purpose of the present disclosure is to provide a task scheduling method, a task scheduling device, a computer-readable storage medium, and an electronic device, thereby overcoming, at least to a certain extent, the problem of low task processing efficiency caused by the limitations and defects of related technologies.

[0008] According to one aspect of the present disclosure, a task scheduling method is provided, the method comprising:

[0009] Obtaining a blood relationship map of the data processing task, and adjusting the blood relationship map to obtain a one-way chain map of the data processing task;

[0010] Obtaining first resource allocation information of a task node in the one-way chain graph where a data resource required for processing the data processing task is located, and a first running time required for processing a historical task associated with the data processing task;

[0011] predicting a second running time required to process the data processing task based on the first resource allocation information and the first running time;

[0012] According to the second running time, the running order of the data processing tasks in the target task list is adjusted.

[0013] In an exemplary embodiment of the present disclosure, adjusting the blood relationship map to obtain a one-way chain map of the data processing task includes:

[0014] The ring-like structure in the blood relationship map is adjusted to obtain a one-way chain map of the data processing task; wherein the ring-like structure is a ring structure with at least one level.

[0015] In an exemplary embodiment of the present disclosure, when the quasi-ring structure is a ring structure having one level, adjusting the quasi-ring structure in the blood relationship map to obtain a unidirectional chain map of the data processing task includes:

[0016] Based on a preset tree diagram generation model, the ring-like structure in the blood relationship map is adjusted to obtain a one-way chain map of the data processing task.

[0017] In an exemplary embodiment of the present disclosure, the ring-like structure in the blood relationship map is adjusted based on a preset tree diagram generation model to obtain a one-way chain map of the data processing task, including:

[0018] Extracting all nodes and directed edges included in the blood relationship map using a preset tree diagram generation model, and generating a node set based on all the nodes;

[0019] Determine any one node from the node set as a current root node, and calculate the weight value of each current root node according to the weight of the directed edge;

[0020] The current root nodes are sorted according to the weight values, and the current root node with the smallest weight value is used as the starting node and the current root node with the largest weight value is used as the ending node to generate a one-way chain graph of the data processing task.

[0021] In an exemplary embodiment of the present disclosure, when the quasi-ring structure is a ring structure having two levels, the quasi-ring structure in the blood relationship map is adjusted to obtain a unidirectional chain map of the data processing task, including:

[0022] Abstracting the inner-level ring structure in the ring-like structure in the blood relationship map to obtain an inner node, and generating a first intermediate relationship map based on the inner node and the remaining data processing tasks in the blood relationship map;

[0023] Adjusting the ring structure included in the first intermediate relationship graph based on a preset tree graph generation model to obtain a first sub-relationship graph;

[0024] Replacing the inner nodes included in the first sub-relationship graph using the inner-level ring structure to obtain a second intermediate relationship graph;

[0025] Repeat the adjustment steps of the first sub-relationship graph to obtain a second sub-relationship graph, and obtain a one-way chain graph of the data processing task based on the second sub-relationship graph.

[0026] In an exemplary embodiment of the present disclosure, predicting a second running time required to process the data processing task based on the first resource allocation information and the first running time includes:

[0027] Inputting the first resource allocation information, the first running time, and the current time node into a preset time prediction model to obtain a second running time required to process the data processing task;

[0028] The preset time prediction model is obtained by training a network model through historical resource allocation information, historical running time, and historical time nodes for processing the historical tasks.

[0029] In an exemplary embodiment of the present disclosure, adjusting the execution order of the data processing tasks in the target task list according to the second execution time includes:

[0030] generating a current task instruction to be processed according to the second running time and the first resource allocation information;

[0031] The current pending task instruction is added to the current task list to obtain a target task list, and the running order of the data processing task in the target task list is dynamically adjusted according to the third running time of other pending task instructions included in the target task list and the second running time of the current pending task instruction.

[0032] In an exemplary embodiment of the present disclosure, the current pending task instruction is added to the current task list to obtain a target task list, including:

[0033] Generate a first parent node corresponding to the current pending task instruction, and obtain a second parent node included in the current task list;

[0034] Establishing a directed connection between the first parent node and the second parent node; wherein the connection direction of the directed connection is from the first parent node to the second parent node;

[0035] Obtain the second child node corresponding to the second parent node, and based on the position of the first parent node in the second parent node and the second child node, and the position of the second parent node and the second child node in the current task list, add the current pending task instruction to the current task list to obtain the target task list.

[0036] In an exemplary embodiment of the present disclosure, dynamically adjusting the execution order of the data processing tasks in the target task list according to the third execution times of other pending task instructions included in the target task list and the second execution time of the current pending task instruction includes:

[0037] Generate a pending task tree according to the first parent node, the second parent node, and the second child node, and calculate a first height, a second height, and a third height of the first parent node, the second parent node, and the second child node in the pending task tree;

[0038] Calculating the first node out-degree, the second node out-degree, and the third node out-degree of the first parent node, the second parent node, and the second child node in the to-be-processed task tree;

[0039] The running order of the data processing tasks in the target task list is dynamically adjusted according to the node out-degree, height and running time of the first parent node, the second parent node and the second child node.

[0040] In an exemplary embodiment of the present disclosure, dynamically adjusting the execution order of the data processing tasks in the target task list according to the node out-degree, height, and execution time of the first parent node, the second parent node, and the second child node includes:

[0041] Calculating a first priority of the first parent node according to the first node out-degree, the first node height, and the second running time;

[0042] Calculate the second priority and the third priority of the second parent node and the second child node according to the second node out-degree, the third node out-degree, the second height, the third height, and the third running time;

[0043] The execution order of the data processing tasks in the target task list is dynamically adjusted according to the first priority, the second priority, and the third priority.

[0044] In an exemplary embodiment of the present disclosure, the priority is calculated as follows: if the height is smaller, the node out-degree is larger, and the task completion time is shorter, the priority is higher;

[0045] The greater the height, the smaller the node out-degree and the longer the task completion time, the lower the priority.

[0046] In an exemplary embodiment of the present disclosure, the task scheduling method further includes:

[0047] When it is detected that any data processing task in the target task list is completed, the to-be-processed task instruction corresponding to the data processing task in the target task list is deleted, and the data processing task is added to the completed task tree.

[0048] In an exemplary embodiment of the present disclosure, obtaining a blood relationship map of data processing tasks includes:

[0049] The initial data is parsed to obtain data processing tasks included in the initial data and the blood relationship between the data processing tasks, and a blood relationship map of the data processing tasks is generated based on the data processing tasks and the blood relationship.

[0050] According to one aspect of the present disclosure, there is provided a task scheduling device, comprising:

[0051] A blood relationship map adjustment module is used to obtain a blood relationship map of a data processing task and adjust the blood relationship map to obtain a one-way chain map of the data processing task;

[0052] A first runtime acquisition module is configured to acquire first resource allocation information of a task node in the tree relationship graph where the data resource required for processing the data processing task is located, and a first runtime required for processing a historical task associated with the data processing task;

[0053] a runtime prediction module, configured to predict a second runtime required to process the data processing task based on the first resource allocation information and the first runtime;

[0054] An execution sequence adjustment module is used to adjust the execution sequence of the data processing tasks in the target task list according to the second execution time.

[0055] In an exemplary embodiment of the present disclosure, adjusting the blood relationship map to obtain a one-way chain map of the data processing task includes:

[0056] The ring-like structure in the blood relationship map is adjusted to obtain a one-way chain map of the data processing task; wherein the ring-like structure is a ring structure with at least one level.

[0057] In an exemplary embodiment of the present disclosure, when the quasi-ring structure is a ring structure having one level, adjusting the quasi-ring structure in the blood relationship map to obtain a unidirectional chain map of the data processing task includes:

[0058] Based on a preset tree diagram generation model, the ring-like structure in the blood relationship map is adjusted to obtain a one-way chain map of the data processing task.

[0059] In an exemplary embodiment of the present disclosure, the ring-like structure in the blood relationship map is adjusted based on a preset tree diagram generation model to obtain a one-way chain map of the data processing task, including:

[0060] Extracting all nodes and directed edges included in the blood relationship map using a preset tree diagram generation model, and generating a node set based on all the nodes;

[0061] Determine any one node from the node set as a current root node, and calculate the weight value of each current root node according to the weight of the directed edge;

[0062] The current root nodes are sorted according to the weight values, and the current root node with the smallest weight value is used as the starting node and the current root node with the largest weight value is used as the ending node to generate a one-way chain graph of the data processing task.

[0063] In an exemplary embodiment of the present disclosure, when the quasi-ring structure is a ring structure having two levels, the quasi-ring structure in the blood relationship map is adjusted to obtain a unidirectional chain map of the data processing task, including:

[0064] Abstracting the inner-level ring structure in the ring-like structure in the blood relationship map to obtain an inner node, and generating a first intermediate relationship map based on the inner node and the remaining data processing tasks in the blood relationship map;

[0065] Adjusting the ring structure included in the first intermediate relationship graph based on a preset tree graph generation model to obtain a first sub-relationship graph;

[0066] Replacing the inner nodes included in the first sub-relationship graph using the inner-level ring structure to obtain a second intermediate relationship graph;

[0067] Repeat the adjustment steps of the first sub-relationship graph to obtain a second sub-relationship graph, and obtain a one-way chain graph of the data processing task based on the second sub-relationship graph.

[0068] In an exemplary embodiment of the present disclosure, predicting a second running time required to process the data processing task based on the first resource allocation information and the first running time includes:

[0069] Inputting the first resource allocation information, the first running time, and the current time node into a preset time prediction model to obtain a second running time required to process the data processing task;

[0070] The preset time prediction model is obtained by training a network model through historical resource allocation information, historical running time, and historical time nodes for processing the historical tasks.

[0071] In an exemplary embodiment of the present disclosure, adjusting the execution order of the data processing tasks in the target task list according to the second execution time includes:

[0072] generating a current task instruction to be processed according to the second running time and the first resource allocation information;

[0073] The current pending task instruction is added to the current task list to obtain a target task list, and the running order of the data processing task in the target task list is dynamically adjusted according to the third running time of other pending task instructions included in the target task list and the second running time of the current pending task instruction.

[0074] In an exemplary embodiment of the present disclosure, the current pending task instruction is added to the current task list to obtain a target task list, including:

[0075] Generate a first parent node corresponding to the current pending task instruction, and obtain a second parent node included in the current task list;

[0076] Establishing a directed connection between the first parent node and the second parent node; wherein the connection direction of the directed connection is from the first parent node to the second parent node;

[0077] Obtain the second child node corresponding to the second parent node, and based on the position of the first parent node in the second parent node and the second child node, and the position of the second parent node and the second child node in the current task list, add the current pending task instruction to the current task list to obtain the target task list.

[0078] In an exemplary embodiment of the present disclosure, dynamically adjusting the execution order of the data processing tasks in the target task list according to the third execution times of other pending task instructions included in the target task list and the second execution time of the current pending task instruction includes:

[0079] Generate a pending task tree according to the first parent node, the second parent node, and the second child node, and calculate a first height, a second height, and a third height of the first parent node, the second parent node, and the second child node in the pending task tree;

[0080] Calculating the first node out-degree, the second node out-degree, and the third node out-degree of the first parent node, the second parent node, and the second child node in the to-be-processed task tree;

[0081] The running order of the data processing tasks in the target task list is dynamically adjusted according to the node out-degree, height and running time of the first parent node, the second parent node and the second child node.

[0082] In an exemplary embodiment of the present disclosure, dynamically adjusting the execution order of the data processing tasks in the target task list according to the node out-degree, height, and execution time of the first parent node, the second parent node, and the second child node includes:

[0083] Calculating a first priority of the first parent node according to the first node out-degree, the first node height, and the second running time;

[0084] Calculate the second priority and the third priority of the second parent node and the second child node according to the second node out-degree, the third node out-degree, the second height, the third height, and the third running time;

[0085] The execution order of the data processing tasks in the target task list is dynamically adjusted according to the first priority, the second priority, and the third priority.

[0086] In an exemplary embodiment of the present disclosure, the priority is calculated as follows: if the height is smaller, the node out-degree is larger, and the task completion time is shorter, the priority is higher;

[0087] The greater the height, the smaller the node out-degree and the longer the task completion time, the lower the priority.

[0088] In an exemplary embodiment of the present disclosure, the task scheduling apparatus further includes:

[0089] The pending task instruction deletion module deletes the pending task instruction corresponding to the data processing task in the target task list when detecting that any data processing task in the target task list is completed, and adds the data processing task to the completed task tree.

[0090] In an exemplary embodiment of the present disclosure, obtaining a blood relationship map of data processing tasks includes:

[0091] The initial data is parsed to obtain data processing tasks included in the initial data and the blood relationship between the data processing tasks, and a blood relationship map of the data processing tasks is generated based on the data processing tasks and the blood relationship.

[0092] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements any one of the above-mentioned task scheduling methods.

[0093] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0094] processor; and

[0095] a memory for storing executable instructions of the processor;

[0096] The processor is configured to execute any one of the above-mentioned task scheduling methods by executing the executable instructions.

[0097] The present disclosure provides a task scheduling method. On the one hand, the second running time required for processing a data processing task can be predicted based on the first resource allocation information and the first running time. Then, the position of the data processing task in the target task list is adjusted based on the second running time. This solves the problem in the prior art that, due to the inability to predict the running time required for each data processing task in advance, the scheduling task list cannot be dynamically adjusted based on the running time of each task, which leads to the inability to prioritize data tasks with shorter predicted running times, resulting in low task processing efficiency. On the other hand, since the blood relationship map can be adjusted to obtain a one-way chain map of the data processing tasks, the blocking problem of predecessor tasks and successor tasks is alleviated, and the overall time consumption of the data processing tasks is shortened. On the other hand, by obtaining the first resource allocation information of the task node where the data resource required for processing the data processing task is located in the tree relationship map, and the first running time required for processing the historical tasks associated with the data processing task, and then predicting the second running time required for processing the data processing task based on the first resource allocation information and the first running time, the problem of parallel calling of resource allocation information of each data processing task, which leads to low task processing efficiency, can be avoided.

[0098] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0100] Figure 1 The following schematically shows a flowchart of a task scheduling method according to an exemplary embodiment of the present disclosure.

[0101] Figure 2 An example diagram of a blood relationship map according to an example embodiment of the present disclosure is schematically shown.

[0102] Figure 3 A diagram schematically illustrates an example scenario in which a blood relationship graph of a ring structure at one level is adjusted to obtain a one-way chain graph according to an exemplary embodiment of the present disclosure.

[0103] Figure 4A diagram schematically illustrates an example scenario in which a two-level ring-structured blood relationship graph is adjusted to obtain a one-way chain graph according to an exemplary embodiment of the present disclosure.

[0104] Figure 5 A diagram schematically illustrates an example scenario in which a blood relationship graph of a ring structure of multiple levels is adjusted to obtain a one-way chain graph according to an exemplary embodiment of the present disclosure.

[0105] Figure 6 A flowchart of a method for adding the current to-be-processed task instruction to the current task list to obtain a target task list according to an exemplary embodiment of the present disclosure is schematically shown.

[0106] Figure 7 An example diagram of a task tree to be processed according to an example embodiment of the present disclosure is schematically shown.

[0107] Figure 8 A flowchart schematically illustrates a method for dynamically adjusting the running order of the data processing tasks in the target task list according to the third running time of other pending task instructions included in the target task list and the second running time of the current pending task instruction according to an example embodiment of the present disclosure.

[0108] Figure 9 An example diagram schematically illustrates priorities of pending task instructions included in a pending task tree according to an example embodiment of the present disclosure.

[0109] Figure 10 The following schematically shows a flowchart of another task scheduling method according to an exemplary embodiment of the present disclosure.

[0110] Figure 11 A block diagram schematically illustrates a task scheduling device according to an exemplary embodiment of the present disclosure.

[0111] Figure 12 An electronic device for implementing the above-mentioned task scheduling method according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0112] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0113] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0114] Existing scheduling strategies cannot directly meet more specific business needs (such as wanting the last task of a module to be completed as quickly as possible). In actual production environments, multiple modules may have scheduling tasks in the same cluster, and these tasks may have dependencies or parallel relationships. At the same time, unless otherwise specified, tasks without dependencies are usually considered to be parallelizable. They are usually scheduled based on the principles of balancing the load of each node and improving system resources. Dynamic adjustment of the task list cannot be achieved. As a result, when the overall execution time of tasks in each module varies greatly (assuming that the scheduling of the last task of a simple module occurs after the scheduling of the last task of a complex module), the task time of the simple module may approach the task time of the complex module.

[0115] Moreover, the task scheduling method in the prior art simply uses blood relationship to define the data task calling relationship. In complex computing scenarios, circular calling scenarios may occur (for example, task 1 calls task 2, task 2 calls task 3, and task 3 calls task 1), which will lead to resource occupation (competition) problems.

[0116] This exemplary embodiment first provides a method for generating a task scheduling list, which can be run on a server, server cluster, or cloud server, etc. Of course, those skilled in the art can also run the method disclosed in this disclosure on other platforms as needed, and this exemplary embodiment does not specifically limit this. Figure 1 As shown, the method for generating the task scheduling list may include the following steps:

[0117] Step S110: Obtain a blood relationship map of the data processing task, and adjust the blood relationship map to obtain a one-way chain map of the data processing task;

[0118] Step S120. Obtain first resource allocation information of a task node in the one-way chain graph where the data resource required to process the data processing task is located, and a first running time required to process a historical task associated with the data processing task;

[0119] Step S130. Predicting a second running time required to process the data processing task based on the first resource allocation information and the first running time;

[0120] Step S140: Adjust the running order of the data processing tasks in the target task list according to the second running time.

[0121] In the above-mentioned task scheduling method, on the one hand, since the second running time required for processing the data processing task can be predicted based on the first resource allocation information and the first running time; and then the position of the data processing task in the target task list is adjusted according to the second running time, thereby solving the problem in the prior art that the running time required for each data processing task cannot be predicted in advance, and therefore the scheduling task list cannot be dynamically adjusted according to the running time of each task, which in turn leads to the inability to prioritize data tasks with shorter predicted running times, resulting in low task processing efficiency, thereby improving task processing efficiency; on the other hand, since the blood relationship map can be adjusted to obtain a one-way chain map of the data processing task, the blocking problem of the predecessor task and the successor task is alleviated, and the overall time consumption of the data processing task is shortened; on the other hand, by obtaining the first resource allocation information of the task node where the data resource required for processing the data processing task is located in the tree relationship map, and the first running time required for processing the historical task associated with the data processing task; and then predicting the second running time required for processing the data processing task based on the first resource allocation information and the first running time, it is possible to avoid the problem of parallel calls of the resource allocation information of each data processing task, thereby resulting in low task processing efficiency.

[0122] Hereinafter, the method for generating a task scheduling list according to an exemplary embodiment of the present disclosure will be explained and illustrated in detail with reference to the accompanying drawings.

[0123] First, the terms involved in the exemplary embodiments of the present disclosure are explained and illustrated.

[0124] The Prophet model can be used to handle situations where time series have some outliers, can also handle situations where some missing values exist, and can almost automatically predict the future trend of time series; the Prophet model can be fitted based on time series decomposition and machine learning.

[0125] Secondly, the invention purpose and application scenario of the example embodiment of the present disclosure are explained and illustrated. Specifically, the method for generating a task scheduling list recorded in the example embodiment of the present disclosure can be applied to a distributed task scheduling scenario with a blood relationship; by considering the data blood relationship between data processing tasks, the blocking problem between the predecessor task and the subsequent task can be alleviated to a certain extent, and the overall time consumption of the task module can be shortened; at the same time, it can also simplify the complex blood relationship graph including the ring or quasi-ring structure, and can generate and dynamically adjust the corresponding task list, to a certain extent, avoid the situation where the actual time consumption of the simple task module is close to the time consumption of the complex task module.

[0126] In a task scheduling method provided in an exemplary embodiment of the present disclosure:

[0127] In step S110, a blood relationship map of the data processing task is obtained, and the blood relationship map is adjusted to obtain a one-way chain map of the data processing task.

[0128] In this example embodiment, first, a blood relationship map of data processing tasks is obtained. Specifically, this can be achieved in the following way: the initial data is parsed to obtain the data processing tasks included in the initial data and the blood relationship between the data processing tasks, and based on the data processing tasks and the blood relationship, a blood relationship map of the data processing tasks is generated. For example, the initial data can be obtained from a cluster (such as a Hadoop cluster, a Hive cluster or MySQL), wherein the initial data can be, for example, SQL code; then, the initial data is parsed by a preset parser (such as a Queryparser parser) to obtain the data processing tasks and the blood relationship between the data processing tasks; further, based on the blood relationship, a blood relationship map is generated; wherein the generated blood relationship map can refer to Figure 2 It should be noted that, during the parsing of the initial data by the Queryparser parser, Queryparser can convert the input SQL code into data processing tasks, and then determine the blood relationship based on the resources that each data processing task depends on.

[0129] Secondly, the blood relationship map is adjusted to obtain a tree relationship map of the data processing task. Specifically, this may include: adjusting the quasi-ring structure in the blood relationship map to obtain a unidirectional chain map of the data processing task; wherein the quasi-ring structure is a ring structure with at least one level, that is, the quasi-ring structure may include any one of a ring structure with one level, a ring structure with two levels, and a ring structure with multiple levels.

[0130] Wherein, when the quasi-ring structure is a ring structure with one level, the quasi-ring structure in the blood relationship map is adjusted to obtain a one-way chain map of the data processing task, including: adjusting the quasi-ring structure in the blood relationship map based on a preset tree graph generation model to obtain a one-way chain map of the data processing task. Wherein, adjusting the quasi-ring structure in the blood relationship map based on a preset tree graph generation model to obtain a one-way chain map of the data processing task can be achieved as follows: first, using a preset tree graph generation model to extract all nodes and directed edges included in the blood relationship map, and generating a node set based on all the nodes; second, determining any one node from the node set as the current root node, and calculating the weight value of each current root node based on the weight of the directed edge; finally, sorting each current root node according to the weight value, and taking the current root node with the smallest weight value as the starting node and the current root node with the largest weight value as the ending node to generate a one-way chain map of the data processing task. The preset tree diagram generation model can be the Zhu-Liu / Edmonds algorithm model, which is an algorithm model for finding the minimum spanning tree diagram of a directed graph. Figure 3 As shown, there are three tasks in the circular task, among which B needs to depend on task A, and C needs to depend on tasks A and B. Then the circular structure can be adjusted to B depending on task A and C depending on task B. It should be supplemented here that the one-way chain graph recorded in the exemplary embodiment of the present disclosure means that in the graph corresponding to the data processing task obtained in the end, the graph is a directed graph, which can be presented in the form of a chain and is a one-way single chain graph. Based on this method, during the execution of data processing tasks, they can be executed in sequence according to the order of precedence, thereby avoiding problems such as resource conflicts.

[0131] Furthermore, when the ring-like structure is a ring structure with two levels, the ring-like structure in the blood relationship map is adjusted to obtain a one-way chain map of the data processing task, including: first, abstracting the inner-level ring structure in the ring-like structure in the blood relationship map to obtain an inner node, and generating a first intermediate relationship map based on the inner node and the remaining data processing tasks in the blood relationship map; secondly, adjusting the ring structure included in the first intermediate relationship map based on a preset tree diagram generation model to obtain a first sub-relationship map; then, using the inner-level ring structure to replace the inner node included in the first sub-relationship map to obtain a second intermediate relationship map; finally, repeating the adjustment step of the first sub-relationship map to obtain a second sub-relationship map, and obtaining a one-way chain map of the data processing task based on the second sub-relationship map. Specifically, refer to Figure 4 As shown, if there are two quasi-ring structures in the blood relationship map, the non-outermost layer ring will be shrunk into a circle, and the outermost layer quasi-ring structure will be adjusted. After the adjustment is completed, the non-outermost layer quasi-ring structure will be restored and adjusted again. The adjustment will be repeated until the quasi-ring structure no longer exists.

[0132] Furthermore, when the quasi-ring structure is a ring structure with multiple levels, the quasi-ring structure in the blood relationship map is adjusted to obtain a one-way chain map of the data processing task, including: first, abstracting the ring structures of all inner levels except the outermost level in the quasi-ring structure in the blood relationship map to obtain an inner node, and generating a first intermediate relationship map based on the inner node and the remaining data processing tasks in the blood relationship map; secondly, adjusting the first intermediate relationship map to a first sub-relationship map based on the blood relationship between the data processing tasks and the inner nodes included in the first intermediate relationship map; then, using the ring structures of all inner levels except the outermost level to replace the inner nodes included in the first sub-relationship map, obtain a second intermediate relationship map; finally, repeating the adjustment steps of the first sub-relationship map until all the ring structures are adjusted to obtain other sub-relationship maps, and obtaining the one-way chain map of the data processing task based on the sub-relationship map adjusted for the last time. Specifically, refer to Figure 5 As shown, if there are multiple complex ring-like structures in the blood relationship map, the non-outermost layer rings will be shrunk into a circle, and the outermost layer ring-like structure will be adjusted. After the adjustment is completed, the non-outermost layer ring-like structure will be restored and adjusted again. The adjustment will be repeated until there is no ring-like structure on the page.

[0133] In step S120, first resource allocation information of the task node where the data resources required to process the data processing task are located in the one-way chain graph, and a first running time required to process the historical tasks associated with the data processing task are obtained.

[0134] Specifically, after obtaining the one-way chain graph, the first resource allocation information of the task node where the data resources required to process the data processing task are located in the one-way chain graph, as well as the first running time required to process the historical tasks associated with the data processing task can be obtained; wherein, the first resource allocation information can be the occupancy status of the resources required to process the data processing task and the number of resources required, etc. The historical task can be a historical processed task with the same task type or the same data type as the data processing task; of course, in a periodic task processing scenario, the historical task can also be a historical processed task that is the same as the data processing task and has been processed in the previous time period. This example does not impose any special restrictions on this.

[0135] In step S130 , a second running time required to process the data processing task is predicted based on the first resource allocation information and the first running time.

[0136] In this example embodiment, first, based on the first resource allocation information and the first runtime, a second runtime required to process the data processing task is predicted. Specifically, this may include inputting the first resource allocation information, the first runtime, and the current time node into a preset time prediction model to obtain the second runtime required to process the data processing task; wherein the preset time prediction model is obtained by training a network model using historical resource allocation information, historical runtimes, and historical time nodes of processing the historical tasks.

[0137] Specifically, the network model can be a Prophet model, and of course it can also be other network models with time prediction functions, such as an auto-regressive moving average model (ARMA model), an exponential smoothing model, etc. This example does not impose special restrictions on this. Furthermore, since the Prophet model can be used to process situations where there are some outliers in the time series; therefore, in the process of predicting the second running time, taking the data calculation task as an example, the daily level of data usually has periodicity. Special activity days such as holidays and version days may cause a surge in data, and the corresponding calculation time may increase accordingly. Therefore, when using the algorithm to predict the task completion time, in addition to holidays, the special dates set in the task corresponding system should also be considered, thereby achieving the purpose of improving the accuracy of the second running time. At the same time, in the process of training the Prophet model, the periodicity problem is also fully taken into account, thereby improving the accuracy of the second running time on the basis of improving the accuracy of the time prediction model.

[0138] Secondly, after obtaining the second running time, the current pending task instructions can be generated based on the second running time and the first resource allocation information, so as to dynamically adjust the task list according to the second running time included in the current pending task instructions, and mobilize the corresponding resources according to the first resource allocation information to process the data processing task.

[0139] In step S140 , the running order of the data processing tasks in the target task list is adjusted according to the second running time.

[0140] In this example embodiment, first, a current pending task instruction is generated according to the second running time and the first resource allocation information; secondly, the current pending task instruction is added to the current task list to obtain a target task list, and the running order of the data processing task in the target task list is dynamically adjusted according to the third running time of other pending task instructions included in the target task list and the second running time of the current pending task instruction. Specifically, in an actual application scenario, when it is necessary to execute one or more data processing tasks, a current pending task instruction corresponding to the data processing task can be generated according to the second running time and the first resource allocation information, and then the current pending task instruction is added to the current task list to obtain a target task list; wherein, in the current task list, there may be other unexecuted pending task instructions and / or pending task instructions being executed, each pending task instruction includes the running time required to execute the data processing task corresponding to the pending task instruction (the third running time, which is also predicted by a prediction method similar to the second running time) and the second resource allocation information required to process the data processing task corresponding to the pending task instruction. At the same time, after obtaining the current pending task instruction, it can be added to the current task list. During the adding process, it is achieved by adding one or more current pending task instructions to the current task list.

[0141] Specifically, refer to Figure 6 As shown, adding the current pending task instruction to the current task list to obtain the target task list may include the following steps:

[0142] Step S610: Generate a first parent node corresponding to the current task instruction to be processed, and obtain a second parent node included in the current task list;

[0143] Step S620: Establish a directed connection between the first parent node and the second parent node; wherein the connection direction of the directed connection is from the first parent node to the second parent node;

[0144] Step S630, obtain the second child node corresponding to the second parent node, and based on the position of the first parent node in the second parent node and the second child node, and the position of the second parent node and the second child node in the current task list, add the current pending task instruction to the current task list to obtain the target task list.

[0145] The following, combined Figure 7 Steps S610 to S630 are explained and illustrated. Figure 7As shown, a new first parent node corresponding to the current pending task instruction is added (for example Figure 7 Then, in the original parent node (the second parent node, for example Figure 7 A connection is established between the parent nodes A, B, C, and D shown in the figure and the newly added parent node (the first parent node), with the direction being from the new parent node (the first parent node) to the original parent node (the second parent node); further, the second child node included in the second parent node is obtained, and based on the position of the first parent node in the second parent node and the second child node, and the position of the second parent node and the second child node in the current task list, the current pending task instruction is added to the current task list.

[0146] , get the target task list. For example, refer to Figure 7 As shown, for example, the first parent node O is located at

[0147] The root node among the second parent nodes A, B, C, D and the second child nodes E, F, G, H, I, J

[0148] The position of the second parent node A, B, C, D and the second child node E, F, G, H, I, J in the current task list is determined, for example, they are in a certain column or a certain row or a certain specific position, and then the first parent node can be added to the column or the row or the specific position.

[0149] Secondly, based on the position of the first parent node in the second parent node and the second child node, and the position of the second parent node and the second child node in the current task list, the current pending task instruction is added to the current task list to obtain the target task list. Figure 8 As shown, the following steps may be included:

[0150] Step S810: generating a pending task tree according to the first parent node, the second parent node, and the second child node, and calculating a first height, a second height, and a third height of the first parent node, the second parent node, and the second child node in the pending task tree;

[0151] Step S820, calculating the first node out-degree, the second node out-degree, and the third node out-degree of the first parent node, the second parent node, and the second child node in the to-be-processed task tree;

[0152] Step S830: Dynamically adjust the running order of the data processing tasks in the target task list according to the node out-degree, height, and running time of the first parent node, the second parent node, and the second child node.

[0153] According to the node out-degree, height and running time of the first parent node, the second parent node and the second child node, the running order of the data processing task in the target task list is dynamically adjusted. Specifically, it can include: first, according to the first node out-degree, the first node height and the second running time, the first priority of the first parent node is calculated; second, according to the second node out-degree, the third node out-degree, the second height, the third height and the third running time, the second priority and the third priority of the second parent node and the second child node are calculated; finally, according to the first priority, the second priority and the third priority, the running order of the data processing task in the target task list is dynamically adjusted. The calculation rules of the first priority, the second priority and the third priority are: if the height is smaller, the node out-degree is larger and the task completion time is shorter, the priority is higher; if the height is larger, the node out-degree is smaller and the task completion time is longer, the priority is lower.

[0154] The following will explain and illustrate steps S810 to S840. Specifically, first, a task tree to be processed is generated based on the first parent node, the second parent node and the second child node. The task tree to be processed can be specifically referred to Figure 7 As shown, the node out-degree (first node out-degree, second node out-degree and third node out-degree) of all nodes included in the pending task tree and the height (first height, second height and third height) of all nodes in the pending task tree are calculated; for example, Figure 7 In the figure, the first node out-degree of node O is 4, the second node out-degree of node A is 2, and the third node out-degree of node F is 1; the second height of node B is 1; further, after obtaining the node height and node out-degree, the pending task instructions in the target task list can be prioritized according to the reverse order principle of the subtree height corresponding to the node (from large to small), the positive order principle of the node out-degree of each node (from small to large), and the positive order rule of the task completion time (from small to large), and then the list of tasks to be scheduled can be obtained.

[0155] It should be noted that in the priority calculation method described in the exemplary embodiment of the present disclosure, the higher the subtree corresponding to the default node, the more complex the dependency relationship of the module and the more complex the intermediate relationship; the larger the node out-degree, the higher the task coreness and the greater the parallelism after the task is completed; the shorter the task completion time, the faster the resources occupied by the task can be released. Therefore, the node with a smaller height, a larger node out-degree and a shorter running time is more critical, and the priority execution of the node can improve the resource utilization of the system; the specific priority sorting results can be referred to Figure 9 ( Figure 9The priority in the table represents the priority ranking of tasks within the queued task list. AC indicates that the task is still executing, and BD indicates that the task has completed execution. Of course, in actual applications, other priority calculation rules can be configured as needed; this example does not specifically limit this.

[0156] At this point, the process of scheduling pending tasks has been completed. Furthermore, after the generation of the pending task list is completed, the pending task instructions included in the target task list can be processed, and when it is detected that any data processing task in the target task list is completed, the pending task instruction corresponding to the data processing task in the target task list is deleted, and the data processing task is added to the completed task tree. In the specific application process, the sorted tasks can be added to the pending task list to wait for scheduling; then, the execution of the pending tasks is monitored, and the queued task list and the pending task list are updated. When a task is completed, the queued task list is updated and the child node of the completed task node is added; if there are tasks in the updated queued list that have not been scheduled, they will participate in the new round of task scheduling and be re-sorted, so as to realize dynamic adjustment of the tasks to be scheduled; when a task starts to be scheduled, the pending task list and the queued task list are updated, and the task that starts scheduling is removed.

[0157] The following, combined Figure 10 The task scheduling method of the exemplary embodiment of the present disclosure is further explained. Specifically, refer to Figure 10 As shown, the task scheduling method may include the following steps:

[0158] Step S1001: Parse the SQL code to obtain a data lineage relationship graph of the data computing tasks in the entire scheduling plan;

[0159] Step S1002: Adjust the ring-like structure (ring structure) included in the blood relationship map to obtain a tree relationship map of the data processing task;

[0160] Step S1003: Generate and dynamically adjust the target task list to obtain a list of tasks to be scheduled;

[0161] Step S1004, determining whether all pending task instructions included in the pending task list are completed; if so, ending; if not, jumping to step S1003.

[0162] The method for scheduling tasks provided by the example embodiments of the present disclosure can adjust the ring structure and ring-like structure between tasks, streamline the dependencies between tasks, and provide support for the calculation and dynamic update of the priorities of subsequent tasks; at the same time, it can predict the task completion time, taking into account the periodic characteristics of data task data (such as holidays, system-specific version dates, etc.) and system resources (such as expansion, etc.), and predict the task completion time at each scheduling as accurately as possible, providing strong support for the calculation and dynamic update of the priorities of subsequent tasks; further, it generates and dynamically adjusts the list of tasks to be scheduled, associates the priority of the task with the out-degree of the task, the predicted completion time of the task, and the height of the corresponding subtree, and continuously adjusts the list of tasks to be scheduled according to the execution status of the task, ensuring the effective use of system resources while meeting business needs.

[0163] The present disclosure also provides a task scheduling device. Figure 11 As shown, the task scheduling device may include a blood relationship map adjustment module 1110, a first running time acquisition module 1120, a running time prediction module 1130, and a running order adjustment module 1140. Among them:

[0164] The blood relationship map adjustment module 1110 can be used to obtain the blood relationship map of the data processing task and adjust the blood relationship map to obtain a one-way chain map of the data processing task;

[0165] The first runtime acquisition module 1120 may be configured to acquire first resource allocation information of a task node in the tree relationship graph where the data resource required for processing the data processing task is located, and a first runtime required for processing a historical task associated with the data processing task;

[0166] A runtime prediction module 1130 may be configured to predict a second runtime required to process the data processing task based on the first resource allocation information and the first runtime;

[0167] The running order adjustment module 1140 may be configured to adjust the running order of the data processing tasks in the target task list according to the second running time.

[0168] In the above-mentioned task scheduling device, on the one hand, since the second running time required for processing the data processing task can be predicted based on the first resource allocation information and the first running time; and then the position of the data processing task in the target task list is adjusted according to the second running time, thereby solving the problem in the prior art that the running time required for each data processing task cannot be predicted in advance, and therefore the scheduling task list cannot be dynamically adjusted according to the running time of each task, which in turn leads to the inability to prioritize data tasks with shorter predicted running times, resulting in low task processing efficiency, thereby improving task processing efficiency; on the other hand, since the blood relationship map can be adjusted to obtain a one-way chain map of the data processing task, the blocking problem of the predecessor task and the successor task is alleviated, and the overall time consumption of the data processing task is shortened; on the other hand, by obtaining the first resource allocation information of the task node where the data resource required for processing the data processing task is located in the tree relationship map, and the first running time required for processing the historical task associated with the data processing task; and then predicting the second running time required for processing the data processing task based on the first resource allocation information and the first running time, it is possible to avoid the problem of parallel calls of the resource allocation information of each data processing task, thereby resulting in low task processing efficiency.

[0169] In an exemplary embodiment of the present disclosure, adjusting the blood relationship map to obtain a one-way chain map of the data processing task includes:

[0170] The ring-like structure in the blood relationship map is adjusted to obtain a one-way chain map of the data processing task; wherein the ring-like structure is a ring structure with at least one level.

[0171] In an exemplary embodiment of the present disclosure, when the quasi-ring structure is a ring structure having one level, adjusting the quasi-ring structure in the blood relationship map to obtain a unidirectional chain map of the data processing task includes:

[0172] Based on a preset tree diagram generation model, the ring-like structure in the blood relationship map is adjusted to obtain a one-way chain map of the data processing task.

[0173] In an exemplary embodiment of the present disclosure, the ring-like structure in the blood relationship map is adjusted based on a preset tree diagram generation model to obtain a one-way chain map of the data processing task, including:

[0174] Extracting all nodes and directed edges included in the blood relationship map using a preset tree diagram generation model, and generating a node set based on all the nodes;

[0175] Determine any one node from the node set as a current root node, and calculate the weight value of each current root node according to the weight of the directed edge;

[0176] The current root nodes are sorted according to the weight values, and the current root node with the smallest weight value is used as the starting node and the current root node with the largest weight value is used as the ending node to generate a one-way chain graph of the data processing task.

[0177] In an exemplary embodiment of the present disclosure, when the quasi-ring structure is a ring structure having two levels, the quasi-ring structure in the blood relationship map is adjusted to obtain a unidirectional chain map of the data processing task, including:

[0178] Abstracting the inner-level ring structure in the ring-like structure in the blood relationship map to obtain an inner node, and generating a first intermediate relationship map based on the inner node and the remaining data processing tasks in the blood relationship map;

[0179] Adjusting the ring structure included in the first intermediate relationship graph based on a preset tree graph generation model to obtain a first sub-relationship graph;

[0180] Replacing the inner nodes included in the first sub-relationship graph using the inner-level ring structure to obtain a second intermediate relationship graph;

[0181] Repeat the adjustment steps of the first sub-relationship graph to obtain a second sub-relationship graph, and obtain a one-way chain graph of the data processing task based on the second sub-relationship graph.

[0182] In an exemplary embodiment of the present disclosure, predicting a second running time required to process the data processing task based on the first resource allocation information and the first running time includes:

[0183] Inputting the first resource allocation information, the first running time, and the current time node into a preset time prediction model to obtain a second running time required to process the data processing task;

[0184] The preset time prediction model is obtained by training a network model through historical resource allocation information, historical running time, and historical time nodes for processing the historical tasks.

[0185] In an exemplary embodiment of the present disclosure, adjusting the execution order of the data processing tasks in the target task list according to the second execution time includes:

[0186] generating a current task instruction to be processed according to the second running time and the first resource allocation information;

[0187] The current pending task instruction is added to the current task list to obtain a target task list, and the running order of the data processing task in the target task list is dynamically adjusted according to the third running time of other pending task instructions included in the target task list and the second running time of the current pending task instruction.

[0188] In an exemplary embodiment of the present disclosure, the current pending task instruction is added to the current task list to obtain a target task list, including:

[0189] Generate a first parent node corresponding to the current pending task instruction, and obtain a second parent node included in the current task list;

[0190] Establishing a directed connection between the first parent node and the second parent node; wherein the connection direction of the directed connection is from the first parent node to the second parent node;

[0191] Obtain the second child node corresponding to the second parent node, and based on the position of the first parent node in the second parent node and the second child node, and the position of the second parent node and the second child node in the current task list, add the current pending task instruction to the current task list to obtain the target task list.

[0192] In an exemplary embodiment of the present disclosure, dynamically adjusting the execution order of the data processing tasks in the target task list according to the third execution times of other pending task instructions included in the target task list and the second execution time of the current pending task instruction includes:

[0193] Generate a pending task tree according to the first parent node, the second parent node, and the second child node, and calculate a first height, a second height, and a third height of the first parent node, the second parent node, and the second child node in the pending task tree;

[0194] Calculating the first node out-degree, the second node out-degree, and the third node out-degree of the first parent node, the second parent node, and the second child node in the to-be-processed task tree;

[0195] The running order of the data processing tasks in the target task list is dynamically adjusted according to the node out-degree, height and running time of the first parent node, the second parent node and the second child node.

[0196] In an exemplary embodiment of the present disclosure, dynamically adjusting the execution order of the data processing tasks in the target task list according to the node out-degree, height, and execution time of the first parent node, the second parent node, and the second child node includes:

[0197] Calculating a first priority of the first parent node according to the first node out-degree, the first node height, and the second running time;

[0198] Calculate the second priority and the third priority of the second parent node and the second child node according to the second node out-degree, the third node out-degree, the second height, the third height, and the third running time;

[0199] The execution order of the data processing tasks in the target task list is dynamically adjusted according to the first priority, the second priority, and the third priority.

[0200] In an exemplary embodiment of the present disclosure, the priority is calculated as follows: if the height is smaller, the node out-degree is larger, and the task completion time is shorter, the priority is higher;

[0201] The greater the height, the smaller the node out-degree and the longer the task completion time, the lower the priority.

[0202] In an exemplary embodiment of the present disclosure, the task scheduling apparatus further includes:

[0203] The pending task instruction deletion module deletes the pending task instruction corresponding to the data processing task in the target task list when detecting that any data processing task in the target task list is completed, and adds the data processing task to the completed task tree.

[0204] In an exemplary embodiment of the present disclosure, obtaining a blood relationship map of data processing tasks includes:

[0205] The initial data is parsed to obtain data processing tasks included in the initial data and the blood relationship between the data processing tasks, and a blood relationship map of the data processing tasks is generated based on the data processing tasks and the blood relationship.

[0206] The specific details of each module in the above task scheduling device have been described in detail in the corresponding task scheduling method, so they will not be repeated here.

[0207] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0208] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0209] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0210] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0211] Refer to the following Figure 12 12 is a diagram to describe the electronic device 1200 according to this embodiment of the present disclosure. Figure 12 The electronic device 1200 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0212] like Figure 12 As shown, electronic device 1200 is implemented as a general-purpose computing device. Components of electronic device 1200 may include, but are not limited to, the aforementioned at least one processing unit 1210, the aforementioned at least one storage unit 1220, a bus 1230 connecting various system components (including storage unit 1220 and processing unit 1210), and a display unit 1240.

[0213] The storage unit stores program codes, which can be executed by the processing unit 1210, so that the processing unit 1210 performs the steps described in the "Exemplary Method" section of the present disclosure according to various exemplary embodiments. For example, the processing unit 1210 can perform the following steps: Figure 1Step S110 shown in: obtaining a blood relationship map of the data processing task, and adjusting the blood relationship map to obtain a one-way chain map of the data processing task; step S120: obtaining first resource allocation information of the task node where the data resource required for processing the data processing task is located in the one-way chain map, and a first running time required to process the historical tasks associated with the data processing task; step S130: predicting a second running time required to process the data processing task based on the first resource allocation information and the first running time; step S140: adjusting the running of the data processing task in the target task list based on the second running time.

[0214] The storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 12201 and / or a cache memory unit 12202 , and may further include a read-only memory unit (ROM) 12203 .

[0215] The storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0216] The bus 1230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0217] The electronic device 1200 can also communicate with one or more external devices 1300 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 1200, and / or any device that enables the electronic device 1200 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 1250. Furthermore, the electronic device 1200 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 1260. As shown, the network adapter 1260 communicates with other modules of the electronic device 1200 via a bus 1230. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 1200, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0218] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0219] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the aforementioned methods of this specification. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present disclosure.

[0220] According to an embodiment of the present disclosure, a program product for implementing the above-mentioned method can be a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0221] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0222] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0223] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0224] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0225] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0226] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not invented herein. The specification and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

Claims

1. A task scheduling method, characterized in that: The method comprises: Obtaining a blood relationship graph of the data processing task, and adjusting a quasi-ring structure in the blood relationship graph to obtain a unidirectional chain graph of the data processing task; wherein the quasi-ring structure is a ring structure having at least one level; Obtaining first resource allocation information of a task node where a data resource required for processing the data processing task is located in the one-way chain graph, and a first running time required for processing a historical task associated with the data processing task; predicting a second running time required to process the data processing task based on the first resource allocation information and the first running time; adjusting the running order of the data processing tasks in the target task list according to the second running time; Among them, when the ring-like structure is a ring structure with two levels, the one-way chain graph is obtained in the following manner: abstracting the inner-level ring structure in the ring-like structure in the blood relationship graph to obtain an inner node, and generating a first intermediate relationship graph based on the inner node and the remaining data processing tasks in the blood relationship graph; adjusting the ring structure included in the first intermediate relationship graph based on the preset tree graph generation model to obtain a first sub-relationship graph; using the inner-level ring structure to replace the inner node included in the first sub-relationship graph to obtain a second intermediate relationship graph; repeating the adjustment step of the first sub-relationship graph to obtain a second sub-relationship graph, and obtaining the one-way chain graph of the data processing task based on the second sub-relationship graph.

2. The task scheduling method according to claim 1, characterized in that: When the quasi-ring structure is a ring structure with one level, the quasi-ring structure in the blood relationship map is adjusted to obtain a unidirectional chain map of the data processing task, including: Based on a preset tree diagram generation model, the ring-like structure in the blood relationship map is adjusted to obtain a one-way chain map of the data processing task.

3. The task scheduling method according to claim 2, characterized in that: Adjusting the ring-like structure in the blood relationship map based on a preset tree diagram generation model to obtain a one-way chain map of the data processing task includes: Extracting all nodes and directed edges included in the blood relationship map using a preset tree diagram generation model, and generating a node set based on all the nodes; Determine any one node from the node set as a current root node, and calculate the weight value of each current root node according to the weight of the directed edge; The current root nodes are sorted according to the weight values, and the current root node with the smallest weight value is used as the starting node and the current root node with the largest weight value is used as the ending node to generate a one-way chain graph of the data processing task.

4. The task scheduling method according to claim 1, wherein: Predicting a second running time required to process the data processing task according to the first resource allocation information and the first running time includes: Inputting the first resource allocation information, the first running time, and the current time node into a preset time prediction model to obtain a second running time required to process the data processing task; The preset time prediction model is obtained by training a network model through historical resource allocation information, historical running time, and historical time nodes for processing the historical tasks.

5. The task scheduling method according to claim 1, wherein: Adjusting the running order of the data processing tasks in the target task list according to the second running time includes: generating a current task instruction to be processed according to the second running time and the first resource allocation information; The current pending task instruction is added to the current task list to obtain a target task list, and the running order of the data processing task in the target task list is dynamically adjusted according to the third running time of other pending task instructions included in the target task list and the second running time of the current pending task instruction.

6. The task scheduling method according to claim 5, characterized in that: Add the current pending task instruction to the current task list to obtain a target task list, including: Generate a first parent node corresponding to the current pending task instruction, and obtain a second parent node included in the current task list; Establishing a directed connection between the first parent node and the second parent node; wherein the connection direction of the directed connection is from the first parent node to the second parent node; Obtain the second child node corresponding to the second parent node, and based on the position of the first parent node in the second parent node and the second child node, and the position of the second parent node and the second child node in the current task list, add the current pending task instruction to the current task list to obtain the target task list.

7. The task scheduling method according to claim 6, characterized in that: Dynamically adjusting the execution order of the data processing tasks in the target task list according to the third execution times of other pending task instructions included in the target task list and the second execution time of the current pending task instruction includes: Generate a pending task tree according to the first parent node, the second parent node, and the second child node, and calculate a first height, a second height, and a third height of the first parent node, the second parent node, and the second child node in the pending task tree; Calculating the first node out-degree, the second node out-degree, and the third node out-degree of the first parent node, the second parent node, and the second child node in the to-be-processed task tree; The running order of the data processing tasks in the target task list is dynamically adjusted according to the node out-degree, height and running time of the first parent node, the second parent node and the second child node.

8. The task scheduling method according to claim 7, characterized in that: Dynamically adjusting the execution order of the data processing tasks in the target task list according to the node out-degree, height, and execution time of the first parent node, the second parent node, and the second child node, including: Calculating a first priority of the first parent node according to the first node out-degree, the first node height, and the second running time; Calculate the second priority and the third priority of the second parent node and the second child node according to the second node out-degree, the third node out-degree, the second height, the third height, and the third running time; The execution order of the data processing tasks in the target task list is dynamically adjusted according to the first priority, the second priority, and the third priority.

9. The task scheduling method according to claim 8, characterized in that: The priority calculation rule is: if the height is smaller, the node out-degree is larger and the task completion time is shorter, the priority is higher; The higher the height, the smaller the node out-degree and the longer the task completion time, the lower the priority.

10. The task scheduling method according to claim 1, wherein: The task scheduling method further includes: When it is detected that any data processing task in the target task list is completed, the to-be-processed task instruction corresponding to the data processing task in the target task list is deleted, and the data processing task is added to the completed task tree.

11. The task scheduling method according to claim 1, wherein: Obtain a bloodline relationship graph of data processing tasks, including: The initial data is parsed to obtain data processing tasks included in the initial data and the blood relationship between the data processing tasks, and a blood relationship map of the data processing tasks is generated based on the data processing tasks and the blood relationship.

12. A task scheduling device, characterized in that: include: A blood relationship map adjustment module is used to obtain a blood relationship map of a data processing task and adjust the quasi-ring structure in the blood relationship map to obtain a unidirectional chain map of the data processing task; wherein the quasi-ring structure is a ring structure having at least one level; A first runtime acquisition module is configured to acquire first resource allocation information of a task node in the one-way chain graph where data resources required for processing the data processing task are located, and a first runtime required for processing historical tasks associated with the data processing task; a runtime prediction module, configured to predict a second runtime required to process the data processing task based on the first resource allocation information and the first runtime; an execution sequence adjustment module, configured to adjust the execution sequence of the data processing tasks in the target task list according to the second execution time; Among them, when the ring-like structure is a ring structure with two levels, the one-way chain graph is obtained in the following manner: abstracting the inner-level ring structure in the ring-like structure in the blood relationship graph to obtain an inner node, and generating a first intermediate relationship graph based on the inner node and the remaining data processing tasks in the blood relationship graph; adjusting the ring structure included in the first intermediate relationship graph based on the preset tree graph generation model to obtain a first sub-relationship graph; using the inner-level ring structure to replace the inner node included in the first sub-relationship graph to obtain a second intermediate relationship graph; repeating the adjustment step of the first sub-relationship graph to obtain a second sub-relationship graph, and obtaining the one-way chain graph of the data processing task based on the second sub-relationship graph.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the task scheduling method according to any one of claims 1 to 11 is implemented.

14. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the task scheduling method according to any one of claims 1 to 11 by executing the executable instructions.

Images