Task scheduling method and related equipment

Abstract

One or more embodiments of the invention provide a task scheduling method and related equipment. The method comprises the following steps: generating a task graph according to task information and network topology information of a to-be-allocated task; nodes in the task graph represent to-be-allocated tasks, node values are cluster contribution values of the to-be-allocated tasks, and edges in the task graph represent resource competition relationships of the connected nodes; dividing the to-be-allocated tasks into a plurality of to-be-allocated task sets according to the task graph; the scheduling priority of the to-be-allocated task set is determined according to the cluster value of the to-be-allocated task set, the set value is the sum of cluster contribution values of all to-be-allocated tasks in the to-be-allocated task set, and the tasks in each to-be-allocated task set do not have a resource competition relationship; and according to the scheduling priority of the to-be-allocated task set and the task information of the to-be-allocated tasks in the to-be-allocated task set, generating network configuration of the to-be-allocated tasks so as to schedule the to-be-allocated tasks.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the field of communication technology, and in particular, to a task scheduling method and related equipment. Background Art

[0002] Optical Circuit Switch (OCS) technology is an efficient optical switching technology that aims to directly establish end-to-end optical circuits through optical switches, bypassing the traditional electronic switching process and providing low-latency and high-bandwidth data transmission. With the development of optical fiber communication technology, OCS technology has been widely used in large-scale data centers, long-distance communications and private networks, and is particularly suitable for scenarios that require stable and efficient transmission. Its main advantages include significantly reducing transmission latency, making full use of optical fiber bandwidth, reducing energy consumption and avoiding network congestion.

[0003] However, OCS technology faces two major hardware limitations: first, each port can only establish a unique circuit connection with another port at the same time; second, when the switch matrix is ​​reconfigured, the ports involved will be temporarily unavailable. These hardware constraints cause OCS communication solutions to face resource competition problems, limiting the flexibility and efficiency of the network.

[0004] In the related technology, the task scheduling problem of communication networks usually focuses on the pure electric switching network (Electronic Circuit Switching) scenario. There are significant differences in the task scheduling process between pure electric switching networks and optical circuit switching networks. The task scheduling solution for pure electric switching networks is not applicable to optical circuit switching networks. Specifically, in pure electric switching networks, the processing and forwarding of tasks depend on the computing power of electronic devices and the bandwidth of the network. Therefore, scheduling strategies usually give priority to how to reduce delays and speed up task completion to improve the overall efficiency of the system. However, optical circuit switching networks avoid the bottleneck of electronic processing through optical signal transmission. Therefore, the optimization scheduling strategy aimed at reducing delays and speeding up task completion is no longer applicable to optical circuit switching networks. Summary of the invention

[0005] In view of this, an object of one or more embodiments of the present application is to provide a task scheduling method and related devices to solve the problems raised by the background technology.

[0006] Based on the above purpose, one or more embodiments of the present application provide a task scheduling method, including:

[0007] Generate a task graph according to the task information and network topology information of the task to be assigned; the nodes in the task graph represent the tasks to be assigned, the node values ​​are the cluster contribution values ​​of the tasks to be assigned, and the edges in the task graph represent the resource competition relationship between the connected nodes;

[0008] According to the task graph, the tasks to be assigned are divided into a plurality of task sets to be assigned; the scheduling priority of the task set to be assigned is determined according to the cluster value of the task set to be assigned, the cluster value is the sum of the cluster contribution values ​​of all the tasks to be assigned in the task set to be assigned, and there is no resource competition relationship between the tasks in each task set to be assigned;

[0009] Generate a network configuration of the task to be assigned according to the scheduling priority of the task set to be assigned and the task information of the task to be assigned in the task set to be assigned, so as to schedule the task to be assigned

[0010] Optionally, the cluster contribution value of the task to be assigned is used to quantify the contribution of the scheduling of the task to be assigned to the improvement of cluster performance;

[0011] The cluster contribution value of any task to be assigned is obtained by performing the following steps:

[0012] Determine competing tasks that have a resource competition relationship with the task to be assigned;

[0013] Evaluate the value gain of completing the task to be assigned, where the value gain represents the positive impact of completing the task to be assigned on improving cluster performance;

[0014] Evaluate the value loss of waiting for the competing task, where the value loss indicates the negative impact of the competing task waiting caused by executing the task to be assigned on the cluster performance improvement;

[0015] The cluster contribution value of the task to be assigned is obtained according to the value gain and the value loss.

[0016] Optionally, the cluster contribution value calculation formula is:

[0017]

[0018] Among them, C j Represents the cluster contribution value of the task to be assigned, CV j The value rate of the task to be assigned is determined by dividing the value of the task to be assigned by the time cost of completing the task to be assigned. k represents the value rate of the competitive task, which is determined by dividing the value of the competitive task by the time cost of completing the competitive task. j Represents a set of competing tasks for the task to be assigned.

[0019] Optionally, according to the task graph, dividing the to-be-assigned tasks into a plurality of to-be-assigned task sets comprises iteratively executing the following steps:

[0020] Traverse all undivided tasks to be assigned;

[0021] According to the task graph, determining a first task to be assigned having the largest cluster contribution value among all tasks to be assigned, and all second tasks to be assigned that have no resource competition relationship with the first task to be assigned;

[0022] Constructing a set of tasks to be assigned according to the first tasks to be assigned and the second tasks to be assigned;

[0023] The first task to be assigned and the second task to be assigned are marked as divided.

[0024] Optionally, in the task graph, each node is set with a corresponding weight, and the weight represents the scheduling priority of the task to be assigned corresponding to the node;

[0025] According to the task graph, the to-be-assigned tasks are divided into a plurality of to-be-assigned task sets, and the following steps are also iteratively performed:

[0026] Traverse all undivided tasks to be assigned;

[0027] According to the task graph, determine the first task to be assigned that has the largest product of the cluster contribution value and the corresponding node weight among all the tasks to be assigned, and all the second tasks to be assigned that have no resource competition relationship with the first task to be assigned;

[0028] Constructing a set of tasks to be assigned according to the first tasks to be assigned and the second tasks to be assigned;

[0029] The first task to be assigned and the second task to be assigned are marked as divided.

[0030] Optionally, a task graph is generated according to the task information and network topology information of the task to be assigned, including:

[0031] According to the task information of the task to be assigned, construct nodes in the task graph, each node corresponding to one of the tasks to be assigned;

[0032] According to the network topology information, determine the location of each computing node in the network and generate a network topology map;

[0033] Determine the communication route of the task to be assigned according to the task information of the task to be assigned, and mark it in the network topology diagram;

[0034] In response to determining that the communication routes of any two tasks to be assigned intersect, the nodes corresponding to the two tasks to be assigned in the task graph are connected.

[0035] Optionally, generating a network configuration of the task to be assigned according to the scheduling priority of the task set to be assigned and the task information of the task to be assigned in the task set to be assigned includes:

[0036] According to the scheduling priority of the set of tasks to be assigned, modifying, by the server controller, a differential services code point field of a message corresponding to the task to be assigned in the set of tasks to be assigned, so as to adjust the priority of the task to be assigned;

[0037] Determine the maximum iteration duration of each set of tasks to be assigned and the optical circuit switch cross-matching model according to the task information of the tasks to be assigned in the set of tasks to be assigned;

[0038] The optical circuit switch is configured through an optical circuit switch controller according to the scheduling priority, maximum iteration duration and cross-matching model of the set of tasks to be assigned.

[0039] Based on the same inventive concept, one or more embodiments of the present application further provide a task scheduling device, including:

[0040] A first computing module is configured to generate a task graph according to task information of the task to be assigned and network topology information; the nodes in the task graph represent the task to be assigned, the node values ​​are cluster contribution values ​​of the task to be assigned, and the edges in the task graph represent resource competition relationships between connected nodes;

[0041] The second computing module is configured to divide the to-be-allocated tasks into a plurality of to-be-allocated task sets according to the task graph; the scheduling priority of the to-be-allocated task set is determined according to the cluster value of the to-be-allocated task set, the cluster value being the sum of the cluster contribution values ​​of all the to-be-allocated tasks in the to-be-allocated task set, and there being no resource competition relationship between the tasks in each of the to-be-allocated task sets;

[0042] The configuration module is configured to generate a network configuration of the task to be assigned according to the scheduling priority of the task set to be assigned and the task information of the task to be assigned in the task set to be assigned, so as to schedule the task to be assigned.

[0043] Based on the same inventive concept, one or more embodiments of the present application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a task scheduling method as described in any one of the above items is implemented.

[0044] Based on the same inventive concept, one or more embodiments of the present application further provide a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute any of the task scheduling methods described above.

[0045] As can be seen from the above, one or more embodiments of the present application provide a task graph that is generated based on the task information and network topology information of the tasks to be assigned; the nodes in the task graph represent the tasks to be assigned, the node values ​​are the cluster contribution values ​​of the tasks to be assigned, and the edges in the task graph represent the resource competition relationship between the connected nodes; according to the task graph, the tasks to be assigned are divided into multiple sets of tasks to be assigned; the scheduling priority of the set of tasks to be assigned is determined based on the cluster value of the set of tasks to be assigned, the set value is the sum of the cluster contribution values ​​of all the tasks to be assigned in the set of tasks to be assigned, and there is no resource competition relationship between the tasks in each set of tasks to be assigned; based on the scheduling priority of the set of tasks to be assigned and the task information of the tasks to be assigned in the set of tasks to be assigned, the network configuration of the tasks to be assigned is generated to schedule the tasks to be assigned.

[0046] In the technical solution of this application, the competitive relationship between tasks and the cluster contribution of tasks to the cluster are clearly displayed by constructing a task graph. Based on the task graph, the task set is divided according to the cluster contribution value, and tasks with high cluster contribution value are executed first, which effectively improves the cluster output capacity and maximizes the overall value.

[0047] The task scheduling device, electronic device and computer-readable storage medium provided in the present application can all implement the steps of the task scheduling method, and therefore also have the beneficial effects of the task scheduling method. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flowchart of a task scheduling method according to one or more embodiments of the present application;

[0050] Figure 2 A schematic diagram of the structure of a task scheduling device according to one or more embodiments of the present application;

[0051] Figure 3 A schematic diagram of competition between tasks in an optoelectronic hybrid switching network according to one or more embodiments of the present application;

[0052] Figure 4 A schematic diagram of the effect of a task scheduling strategy in one or more embodiments of the present application;

[0053] FIG5( a ) is a schematic diagram of the working process of the analyzer according to one or more embodiments of the present application;

[0054] FIG5( b ) is a schematic diagram of the working process of the analyzer according to one or more embodiments of the present application;

[0055] Figure 6 A schematic diagram of the hardware structure of an electronic device according to one or more embodiments of the present application. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0057] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present application should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in one or more embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0058] As described in the background art, in the related art, the task scheduling problem of the communication network usually focuses on the pure electronic circuit switching network (Electronic Circuit Switching) scenario.

[0059] However, with the development of science and technology, the internal traffic of communication network data centers has shown explosive growth. The use of 10GigE switching architecture at the access layer and 40G / 100GigE switching architecture at the core layer has basically become the future development trend of data center networks. In this case, the electrical interconnection architecture of the data center will face technical requirements of high bandwidth, large capacity, low overhead and low energy consumption.

[0060] Taking one of the application scenarios in the embodiments of this application as an example, Distributed Machine Learning (DML) is a method of using multiple computing devices to collaboratively perform machine learning tasks. Considering the continuous expansion of data volume and the increasing complexity of artificial intelligence models, traditional single-accelerator learning methods cannot cope with it. In related technologies, computing tasks are distributed on multiple accelerators to speed up model training and process large-scale data sets to solve the above problems. In order to further adapt to the requirements of frequent communication of DML technology, parallelization strategies and collective communication algorithms are optimized in related technologies.

[0061] The above parallelization strategies can be divided into data parallelism, model parallelism, and hybrid parallelism, depending on how the tasks are partitioned and distributed. Data parallelism divides a large dataset into multiple subsets, each of which is assigned to a different computing node for parallel processing, thereby improving computing performance. In this framework, all nodes independently perform the same operation or algorithm on their respective data subsets. Subsequently, the final output result is obtained by aggregating the calculation results of each node. Model parallelism is a technique for distributing the components of a neural network model to multiple nodes in order to speed up the training process. It is mainly divided into two methods: tensor parallelism and pipeline parallelism. Tensor parallelism involves splitting the model's tensors (such as weight matrices) according to specific dimensions and assigning these split tensors to multiple nodes for parallel computing. Pipeline parallelism, on the other hand, divides the model into different layers, with each node responsible for executing a different subset of the model layers. By distributing the computational load, model parallelism reduces the resource requirements on a single node, thereby facilitating the training of large-scale models. Hybrid parallelism combines the above parallelization strategies to support the processing of larger datasets and models. However, complex parallel strategies and large-scale distributed computing mean that DML tasks require a large number of nodes to participate in completing complex parameter synchronization processes to ensure the consistency of calculation results, which increases the difficulty of achieving communication between workers.

[0062] Collective communications such as AllReduce, AllGather, Broadcast, and All-to-All are often used to facilitate parameter synchronization between DML workers. These operators are characterized by synchronous communication, in which all workers must complete data transmission, reception, and processing before entering the next communication phase. Represented by Ring-AllReduce, this approach organizes computing nodes into a ring topology, allowing sequential delivery and aggregation of data blocks. During each iteration of data transmission and aggregation, all workers must synchronize and wait for each other to ensure the consistency of the aggregation results. The synchronicity of these collective communication operators requires the network infrastructure to have a stable physical topology and sufficient bandwidth resources, which is critical to mitigate the impact of tail latency, which can significantly degrade collective communication performance.

[0063] In addition, the network performance of the training cluster is a key factor affecting the speed of distributed training. With the improvement of the computing power of hardware accelerators such as GPUs and TPUs, and the continuous expansion of cluster size, network bandwidth has gradually become a bottleneck factor restricting end-to-end training performance.

[0064] At this point, considering that the OCS network optical circuit switch has the characteristics of low latency, low power consumption and rate transparency, the deployment of the OCS network can greatly reduce the construction cost of the data center network. The reconfigurability of the OCS network allows the network to dynamically adjust the physical topology according to the artificial intelligence workload, which is very suitable for DML tasks. Related technologies also propose to introduce the OCS network into the training cluster to make up for the disadvantages of the electrical packet switch in bandwidth and latency. At the same time, the reconfigurable optical interconnection enhances the dynamics of the network topology to cope with changing communication needs. Optical circuit switching technology can well cope with the above challenges. Optical circuit switching technology directly switches the input signal to different output ends in the optical domain through space division, time division or wavelength division without any optical / electrical conversion. Compared with electrical packet switching, optical switching does not require optical / electrical or electrical / optical conversion between the optical fiber transmission line and the switch, and in the switching process, it can also give full play to the advantages of high speed, broadband and no electromagnetic induction of optical signals. Since optical switching does not involve electrical signals, it is not restricted by the processing speed of electronic devices and can match the high-speed optical fiber transmission rate. It is independent of the communication protocol, data format and transmission rate, and can achieve transparent data transmission.

[0065] The optical switching architecture ensures that resources are exclusively allocated to specific DML tasks, and each task independently occupies the required optical circuit connection to avoid mutual interference. This is because the OCS network can only establish one-to-one circuit connections concurrently, and is limited by reconfiguration delays. Static slicing resource allocation can guarantee DML task performance. However, this resource allocation strategy will result in a lot of bandwidth waste. Considering the existing traffic pattern of AI workloads, the burst traffic periodically generated by DML tasks cannot continuously utilize the link bandwidth, resulting in exclusive resource allocation causing a large amount of idle link bandwidth. Multi-task shared link resource allocation can alleviate the problem of idle bandwidth, allowing the cluster to accommodate more computing nodes and tasks, and enhance the scalability of the cluster.

[0066] However, due to the port restrictions of the OCS network, traffic from different tasks cannot share the same port at the same time. For example, in actual application scenarios, given that DML tasks require frequent synchronous communication, it is beneficial to divide hardware with close physical distance into a group of computing nodes when deploying new DML tasks. However, the dynamic arrival and departure of tasks often leads to the dispersion of computing resources within the DML cluster. The utilization of these dispersed resources may lead to a situation where a single DML task is distributed across multiple racks, while multiple tasks are deployed on the same rack. In this case, multiple tasks sharing the same rack cannot establish the required optical path connections at the same time, because the top of rack (ToR) switch can only establish an optical path connection with a unique ToR at any time. This limitation leads to communication contention between tasks. Figure 3 For example, Task 3 shares a ToR switch with Task 1 and Task 2. In particular, the target ToR of Task 3 traffic is different from that of Task 1 and Task 2. Therefore, the OCS network cannot meet the communication needs of all three tasks at the same time, resulting in optical path connection competition between Task 3 and the other two tasks. Communication scheduling is a key issue that must be solved in optical DML clusters.

[0067] However, as described in the background art, current optimization schemes for task scheduling are mainly oriented to pure electric switching network scenarios. The optimization methods proposed in these studies are not necessarily applicable to optical switching networks.

[0068] Specifically, in a purely electrical switching network, the processing and forwarding of tasks depends on the computing power of electronic devices and the bandwidth of the network. Therefore, the scheduling strategy usually prioritizes how to reduce delays and speed up task completion to improve the overall efficiency of the system. However, optical circuit switching networks avoid the bottleneck of electronic processing through optical signal transmission, so the optimization scheduling strategy aimed at reducing delays and speeding up task completion is no longer applicable to optical circuit switching networks.

[0069] From another perspective, existing traffic scheduling methods tailored for optical switching networks do not fully capture the unique characteristics of DML tasks. Although these methods generally aim to optimize the job completion time (JCT), prioritizing only JCT may harm the overall value generated by the DML cluster. In commercial DML clusters, the value hierarchy between tasks greatly affects scheduling efficiency.

[0070] Therefore, this application proposes a task scheduling method to solve the above technical problems. In the technical solution of this application, the competitive relationship between tasks and the cluster contribution of tasks to the cluster are clearly displayed by constructing a task graph. Based on the task graph, the task set is divided according to the cluster contribution value, and tasks with high cluster contribution values ​​are executed first, which effectively improves the cluster output capacity and maximizes the overall value.

[0071] like Figure 4 As shown, it is a test experiment of the technical solution of the present application. The three tasks in the figure represent different types of artificial intelligence model training tasks: Task 1, which represents the large-scale language model training task, has the highest value; Task 2 corresponding to the traditional artificial intelligence model has the second highest value; Task 3 used for experimental testing has the lowest value (that is, in terms of value, Task 1>Task 2>Task 3). When Task 3 is prioritized, it will hinder the optical path connection required by Task 1 and Task 2, causing them to be delayed while waiting for resource availability. However, since Task 1 and Task 2 are deployed on different racks, they do not compete for the same optical path connection. Therefore, prioritizing the concurrent scheduling of Task 1 and Task 2 can maximize the output of the cluster and enhance the overall value generation, thereby highlighting the necessity of customizing optimization strategies according to the specific needs of DML task loads in optical networks.

[0072] The following is a detailed description of the technical solution of this application. Figure 1 The task scheduling method of one or more embodiments of the present application includes the following steps:

[0073] Step S101: Generate a task graph according to the task information and network topology information of the task to be assigned; the nodes in the task graph represent the tasks to be assigned, the node values ​​are the cluster contribution values ​​of the tasks to be assigned, and the edges in the task graph represent the resource competition relationship between the connected nodes.

[0074] Step S102: According to the task graph, the to-be-assigned tasks are divided into a plurality of to-be-assigned task sets; the scheduling priority of the to-be-assigned task set is determined according to the cluster value of the to-be-assigned task set, and the set value is the sum of the cluster contribution values ​​of all to-be-assigned tasks in the to-be-assigned task set, and there is no resource competition relationship between the tasks in each of the to-be-assigned task sets.

[0075] Step S103: Generate a network configuration of the task to be assigned according to the scheduling priority of the task set to be assigned and the task information of the task to be assigned in the task set to be assigned, so as to schedule the task to be assigned.

[0076] In order to maximize the cluster value, this application creatively proposes a "Maximum Value Contribution First (MVCF)" scheduling algorithm. The core principle of the MVCF scheduling algorithm is to divide all tasks into several groups to ensure that there is no competition within each group; each group is regarded as a whole to calculate its cluster contribution value, and the scheduling priority is determined based on its cluster contribution value. Considering that the search space for group division is huge, and the relationship between tasks will make this problem more complicated. Step S101 of this application first constructs a task graph for the tasks to be assigned based on the above principles to solve the above problems.

[0077] That is, in the technical solution of the present application, the determination of the priority of the task to be assigned mainly depends on the cluster contribution value of the task to be assigned and the resource conflict relationship with other tasks to be assigned. The task graph in step S101 is intended to show the resource competition relationship of all tasks to be assigned and the cluster contribution value of the task, which serves as the basis for assigning the priority of the task.

[0078] In an embodiment of the present application, a method for constructing a task graph may include: constructing nodes in the task graph according to the task information of the tasks to be assigned, each node corresponding to one of the tasks to be assigned; determining the location of each computing node in the network and generating a network topology graph according to the network topology information; determining the communication route of the tasks to be assigned according to the task information of the tasks to be assigned, and marking it in the network topology graph; in response to determining that the communication routes of any two tasks to be assigned intersect, connecting the nodes corresponding to the two tasks to be assigned in the task graph.

[0079] In addition to constructing the topological structure of the task graph, the present application also uses node values ​​to represent the contribution value of the task to be assigned to the cluster. Specifically, in the embodiment of the present application, the contribution value of the task to be assigned can be obtained through the following steps: determining the competing tasks that have a resource competition relationship with the task to be assigned; evaluating the value gain of completing the task to be assigned, and the value gain represents the positive impact of completing the task to be assigned on the cluster performance improvement; evaluating the value loss of waiting for the competing task, and the value loss represents the negative impact of the waiting of the competing task caused by executing the task to be assigned on the cluster performance improvement; according to the value gain and the value loss, the cluster contribution value of the task to be assigned is obtained.

[0080] Furthermore, in the embodiment of the present application, the cluster contribution value calculation formula may be:

[0081]

[0082] Among them, C j Represents the cluster contribution value of the task to be assigned, CV j The value rate of the task to be assigned is determined by dividing the value of the task to be assigned by the time cost of completing the task to be assigned. k represents the value rate of the competitive task, which is determined by dividing the value of the competitive task by the time cost of completing the competitive task. j Represents a set of competing tasks for the task to be assigned.

[0083] It can be seen that when calculating the cluster contribution value of the task to be assigned, the present application needs to consider not only the value that can be brought by completing the task, but also the blocking effect of completing the task on other tasks.

[0084] In the embodiments of the present application, a preset analyzer may be used to construct a task graph.

[0085] As described above, the cluster contribution value of the task to be assigned reflects its own value and also reflects its blocking effect on other tasks, so tasks with higher cluster value contribution are more likely to appear in groups with higher total value. In the embodiment of the present application, the set of tasks to be assigned can be divided based on this principle.

[0086] In step S102, the tasks to be assigned are divided into multiple sets of tasks to be assigned, including iteratively executing the following steps: traversing all undivided tasks to be assigned; determining, according to the task graph, the first task to be assigned with the largest cluster contribution value among all tasks to be assigned, and all second tasks to be assigned that have no resource competition relationship with the first task to be assigned; constructing a set of tasks to be assigned based on the first task to be assigned and the second task to be assigned; marking the first task to be assigned and the second task to be assigned as divided.

[0087] In addition, in the embodiments of the present application, the weight of the node can also be comprehensively considered, and the weight represents the scheduling priority of the task to be assigned corresponding to the node. In this way, according to the task graph, the task to be assigned is divided into multiple sets of tasks to be assigned, and it also includes iteratively executing the following steps: traversing all undivided tasks to be assigned; according to the task graph, determining the first task to be assigned that has the largest product of the cluster contribution value and the corresponding node weight among all tasks to be assigned, and all second tasks to be assigned that do not have a resource competition relationship with the first task to be assigned; constructing a set of tasks to be assigned based on the first task to be assigned and the second task to be assigned; marking the first task to be assigned and the second task to be assigned as divided.

[0088] The above operation aims to treat each group as a whole, calculate its value, and determine the scheduling priority based on its value. In this way, all tasks are divided into several groups, ensuring that there is no competition within each group, and finding the compatible task set with the highest total value.

[0089] The scheduling queue obtained in steps S101-S102 represents the highest level of decision-making results, but does not generate the final network configuration. Therefore, in an embodiment of the present application, a controller is designed to automatically generate the corresponding network device configuration. The above-mentioned controller can be composed of two parts: a server controller that manages the electrical network and an optical circuit switch controller that manages the optical network. The server controller assigns priority to the traffic of each task according to the scheduling queue. Specifically, each set in the scheduling queue is assigned a corresponding priority, which is then shared by all task flows in the set. The optical circuit switch controller can first calculate the optical path connection required for each set in the scheduling queue, and represent the optical circuit switch cross-matching of any set of tasks to be assigned as an optical circuit switch cross-matching model; then, the optical circuit switch controller will set the duration of each optical circuit switch cross-matching model according to the maximum iteration duration in each set. This method reduces the frequency of reconfiguration of the optical circuit switch and minimizes the throughput loss caused by reconfiguration delays.

[0090] In other words, based on the scheduling priority of the set of tasks to be assigned and the task information of the tasks to be assigned in the set of tasks to be assigned, the network configuration of the tasks to be assigned is generated, including: based on the scheduling priority of the set of tasks to be assigned, modifying the differential service code point field of the message corresponding to the tasks to be assigned in the set of tasks to be assigned through a server controller to adjust the priority of the tasks to be assigned; based on the task information of the tasks to be assigned in the set of tasks to be assigned, determining the maximum iteration time of each set of tasks to be assigned and the cross-matching model of the optical circuit switch; based on the scheduling priority, maximum iteration time and cross-matching model of the set of tasks to be assigned, configuring the optical circuit switch through an optical circuit switch controller.

[0091] The present application is further described below with a specific embodiment of the present application. As shown in FIG5(a), six tasks are deployed in the cluster of the embodiment, and the task deployment and network topology are shown in the figure. According to the task deployment, it can be analyzed that there is a resource competition relationship between task 1 and task 4 and task 2, a resource competition relationship between task 2 and task 1, task 3 and task 5, a resource competition relationship between task 3 and task 2, task 5 and task 6, a resource competition relationship between task 4 and task 1 and task 2, a resource competition relationship between task 5 and task 2 and task 3, and a resource competition relationship between task 6 and task 3.

[0092] In the embodiment of the present application, the node position in the task can be determined according to the deployment position of the task, and then the nodes in the task graph can be connected according to the resource competition relationship between the tasks to obtain the task graph topology structure as shown in Figure 5(b). In this embodiment, the cluster contribution value relationship of each node is known through calculation, and C6>C3>C5>C4>C2>C1.

[0093] After that, the MVCF scheduling algorithm proposed in this application can be executed to divide the set of tasks to be assigned. In this embodiment, firstly, by traversing to determine that the cluster contribution value of task 6 is the largest, task 6 is added to the first set of tasks to be assigned; among tasks 1, task 2, task 4 and task 5 that are compatible with task 6, task 5 has the largest cluster contribution value, so task 5 is added to the first set of tasks to be assigned; tasks 1 and task 4 are compatible with both task 6 and task 5, and task 4 with a larger cluster contribution value is added to the first set of tasks to be assigned. Since task 1 is not compatible with task 4, task 1 cannot be added to the first set of tasks to be assigned. Other sets of tasks to be assigned are divided based on the same principle. In this embodiment, three sets of tasks to be assigned are finally obtained: S1{task 4, task 5, task 6}, S2{task 3, task 1} and S3{task 2}.

[0094] In the embodiment of the present application, Task 6, Task 3 and Task 2 are the tasks with the longest iteration period in their respective sets. The OCS optical circuit switch cross-matching models M1, M2 and M3 required for the three task sets can be calculated by the OCS controller, and then M1 and its configuration time (iteration duration of Task 6), M2 and its configuration time (iteration duration of Task 3) and M3 and its configuration time (iteration duration of Task 2) are configured in the OCS so that it can be reconfigured periodically; then different priorities are assigned to the traffic of S1, S2 and S3 respectively through the server controller, and priority division is achieved by modifying the dscp field of the corresponding message.

[0095] It can be understood that the method can be executed by any device, equipment, platform, or device cluster having computing and processing capabilities.

[0096] It should be noted that the method of one or more embodiments of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of one or more embodiments of the present application, and the multiple devices will interact with each other to complete the described method.

[0097] It should be noted that the specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0098] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides a task scheduling device. Figure 2 As shown, the device comprises:

[0099] The first computing module 11 is configured to generate a task graph according to the task information of the task to be assigned and the network topology information; the nodes in the task graph represent the task to be assigned, the node values ​​are the cluster contribution values ​​of the task to be assigned, and the edges in the task graph represent the resource competition relationship between the connected nodes;

[0100] The second computing module 12 is configured to divide the to-be-assigned tasks into a plurality of to-be-assigned task sets according to the task graph; the scheduling priority of the to-be-assigned task set is determined according to the cluster value of the to-be-assigned task set, the cluster value being the sum of the cluster contribution values ​​of all the to-be-assigned tasks in the to-be-assigned task set, and there being no resource competition relationship between the tasks in each of the to-be-assigned task sets;

[0101] The configuration module 13 is configured to generate a network configuration of the task to be assigned according to the scheduling priority of the task set to be assigned and the task information of the task to be assigned in the task set to be assigned, so as to schedule the task to be assigned.

[0102] For the convenience of description, the above devices are described in terms of functions and modules. Of course, when implementing one or more embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0103] The device of the embodiment is used to implement the corresponding method in the aforementioned embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0104] Figure 6 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0105] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0106] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solution provided in the embodiment of the present application is implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0107] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0108] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0109] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0110] It should be noted that, although the device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the device may also only include the components necessary for implementing the embodiment of the present application, and does not necessarily include all the components shown in the figure.

[0111] The electronic device of the embodiment is used to implement the corresponding method in the aforementioned embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0112] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0113] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0114] In addition, to simplify the description and discussion, and in order not to make one or more embodiments of the present application difficult to understand, the known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making one or more embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which one or more embodiments of the present application will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present disclosure, it is obvious to those skilled in the art that one or more embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0115] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0116] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of this disclosure.

Claims

1. A task scheduling method, characterized in that: include: Generate a task graph according to the task information and network topology information of the task to be assigned; the nodes in the task graph represent the tasks to be assigned, the node values ​​are the cluster contribution values ​​of the tasks to be assigned, and the edges in the task graph represent the resource competition relationship between the connected nodes; According to the task graph, the tasks to be assigned are divided into a plurality of task sets to be assigned; the scheduling priority of the task set to be assigned is determined according to the cluster value of the task set to be assigned, the cluster value is the sum of the cluster contribution values ​​of all the tasks to be assigned in the task set to be assigned, and there is no resource competition relationship between the tasks in each task set to be assigned; According to the scheduling priority of the set of tasks to be assigned and the task information of the tasks to be assigned in the set of tasks to be assigned, a network configuration of the tasks to be assigned is generated to schedule the tasks to be assigned.

2. The method according to claim 1, characterized in that The cluster contribution value of the task to be assigned is used to quantify the contribution of the scheduling of the task to be assigned to the improvement of cluster performance; The cluster contribution value of any task to be assigned is obtained by performing the following steps: Determine competing tasks that have a resource competition relationship with the task to be assigned; Evaluate the value gain of completing the task to be assigned, where the value gain represents the positive impact of completing the task to be assigned on improving cluster performance; Evaluate the value loss of waiting for the competing task, where the value loss indicates the negative impact of the competing task waiting caused by executing the task to be assigned on the cluster performance improvement; The cluster contribution value of the task to be assigned is obtained according to the value gain and the value loss.

3. The method according to claim 2, characterized in that The cluster contribution value calculation formula is: Among them, C j Represents the cluster contribution value of the task to be assigned, CV j The value rate of the task to be assigned is determined by dividing the value of the task to be assigned by the time cost of completing the task to be assigned. k represents the value rate of the competitive task, which is determined by dividing the value of the competitive task by the time cost of completing the competitive task. j Represents a set of competing tasks for the task to be assigned.

4. The method according to claim 2 or 3, characterized in that: According to the task graph, the to-be-assigned tasks are divided into a plurality of to-be-assigned task sets, including iteratively executing the following steps: Traverse all undivided tasks to be assigned; According to the task graph, determining a first task to be assigned having the largest cluster contribution value among all tasks to be assigned, and all second tasks to be assigned that have no resource competition relationship with the first task to be assigned; Constructing a set of tasks to be assigned according to the first tasks to be assigned and the second tasks to be assigned; The first task to be assigned and the second task to be assigned are marked as divided.

5. The method according to claim 2 or 3, characterized in that: In the task graph, each node is set with a corresponding weight, which represents the scheduling priority of the task to be assigned corresponding to the node; According to the task graph, the to-be-assigned tasks are divided into a plurality of to-be-assigned task sets, and the following steps are also iteratively performed: Traverse all undivided tasks to be assigned; According to the task graph, determine the first task to be assigned that has the largest product of the cluster contribution value and the corresponding node weight among all the tasks to be assigned, and all the second tasks to be assigned that have no resource competition relationship with the first task to be assigned; Constructing a set of tasks to be assigned according to the first tasks to be assigned and the second tasks to be assigned; The first task to be assigned and the second task to be assigned are marked as divided.

6. The method according to claim 1, characterized in that Generate a task graph based on the task information and network topology information of the task to be assigned, including: According to the task information of the task to be assigned, construct nodes in the task graph, each node corresponding to one of the tasks to be assigned; According to the network topology information, determine the location of each computing node in the network and generate a network topology map; Determine the communication route of the task to be assigned according to the task information of the task to be assigned, and mark it in the network topology diagram; In response to determining that the communication routes of any two tasks to be assigned intersect, the nodes corresponding to the two tasks to be assigned in the task graph are connected.

7. The method according to claim 1, characterized in that Generating a network configuration of the task to be assigned according to the scheduling priority of the task set to be assigned and the task information of the task to be assigned in the task set to be assigned, including: According to the scheduling priority of the set of tasks to be assigned, modifying, by the server controller, a differential services code point field of a message corresponding to the task to be assigned in the set of tasks to be assigned, so as to adjust the priority of the task to be assigned; Determine the maximum iteration duration of each set of tasks to be assigned and the optical circuit switch cross-matching model according to the task information of the tasks to be assigned in the set of tasks to be assigned; The optical circuit switch is configured through an optical circuit switch controller according to the scheduling priority, maximum iteration duration and cross-matching model of the set of tasks to be assigned.

8. A task scheduling device, characterized in that: include: A first computing module is configured to generate a task graph according to task information and network topology information of the task to be assigned; The nodes in the task graph represent tasks to be assigned, the node values ​​are cluster contribution values ​​of the tasks to be assigned, and the edges in the task graph represent resource competition relationships between connected nodes; The second computing module is configured to divide the to-be-allocated tasks into a plurality of to-be-allocated task sets according to the task graph; the scheduling priority of the to-be-allocated task set is determined according to the cluster value of the to-be-allocated task set, the cluster value being the sum of the cluster contribution values ​​of all the to-be-allocated tasks in the to-be-allocated task set, and there being no resource competition relationship between the tasks in each of the to-be-allocated task sets; The configuration module is configured to generate a network configuration of the task to be assigned according to the scheduling priority of the task set to be assigned and the task information of the task to be assigned in the task set to be assigned, so as to schedule the task to be assigned.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute any one of claims 1 to 7.

Images