Multi-task collaborative traffic scheduling system and method based on port bandwidth reservation mode

By adopting the port bandwidth reservation mode of multi-task collaborative traffic scheduling system in distributed computing clusters, combined with the flow table generation and dynamic resource adjustment of SDN switches, the problem of low network bandwidth resource scheduling efficiency in multi-task scenarios is solved, and the network bandwidth resource scheduling efficiency is realized Efficient resource utilization and resource allocation for high-priority tasks.

CN117560331BActive Publication Date: 2024-11-08ZHEJIANG LAB
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202311453851.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-03
Publication Date
2024-11-08
Estimated Expiration
2043-11-03

AI Technical Summary

Technical Problem

In multi-task scenarios, it is difficult for distributed computing clusters to effectively schedule and optimize network bandwidth resources, resulting in low overall efficiency and untimely allocation of high-priority tasks. The existing technology fails to effectively improve the overall efficiency and bandwidth resource utilization rate of multi-tasks. .

Method used

Adopt a multi-task collaborative traffic scheduling system based on port bandwidth reservation mode. Through task-level resource allocation and traffic-level resource allocation, combined with the flow table generation and resource dynamic adjustment of SDN switches, the task traffic scheduling is optimized and the preemption of high-priority tasks is achieved. and design resource change notification mechanism.

Benefits of technology

The overall task time and single task time in multi-task scenarios are improved, the utilization rate of the cluster network is improved, more task needs are met, and the packet loss rate of task traffic is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117560331B_ABST
    Figure CN117560331B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-task collaborative traffic scheduling system and method based on a port bandwidth reservation mode. In a multi-user multi-task scenario, collaborative traffic scheduling simulation task parameters and simulation task service quality requirements are obtained, and a two-level bandwidth resource allocation strategy is formulated according to the volume, characteristics and service requirements of each task. The scopes of the two-level strategies are respectively the job task-level resource allocation and the job traffic-level resource allocation. End-to-end bandwidth resource reservation is achieved through an SDN (Software Defined Network) switch, thereby ensuring that the bandwidth, delay, packet loss rate, etc. during the execution of different tasks meet the service requirements, while reducing the overall time of multiple tasks to achieve global optimization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of collaborative traffic scheduling, and in particular, relates to a multi-task collaborative traffic scheduling system and method based on a port bandwidth reservation mode. Background Art

[0002] Network services can generally be divided into real-time services and non-real-time services based on real-time requirements. Among them, real-time services are more sensitive to changes in network quality and have higher requirements for network stability, such as real-time big data processing services, high-performance computing services, large model training services, and voice and video call services. Especially with the increasing popularity of cloud computing platforms, it is not uncommon for the same computing cluster to handle multiple tenants and multiple services. The cost of providing each computing task with an independent operating environment to ensure efficient execution of computing tasks is becoming increasingly high, and it is also full of challenges to the overall operating efficiency of the cluster environment.

[0003] For a distributed computing cluster, the scheduling goal is not only to improve the execution efficiency of a single task, but also to improve the overall efficiency of multiple tasks and bandwidth resource utilization. In addition, the management system of the distributed computing cluster should also consider the processing of high-priority computing tasks, and give priority to providing computing and network resources for high-priority tasks on the basis of reducing packet loss of other tasks. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies in the prior art and to propose a multi-task collaborative traffic scheduling system and method based on a port bandwidth reservation mode.

[0005] The present invention is achieved through the following technical solutions:

[0006] A first aspect of an embodiment of the present invention provides a multi-task collaborative traffic scheduling method based on a port bandwidth reservation mode, the method comprising:

[0007] Obtain collaborative traffic scheduling simulation task parameters and simulation task service quality requirements;

[0008] When the collaborative traffic scheduling simulation task is started, the task bandwidth resource application is performed according to the simulation task parameters and the simulation task service quality requirements, and the network bandwidth resource pool of the collaborative traffic scheduling simulation task is obtained to complete the task-level resource allocation;

[0009] When the collaborative traffic scheduling simulation task is started or any collaborative traffic transmission subtask is completed, according to the dependency between the calculation phase and the transmission phase in the simulation task and the corresponding characteristics of the transmission phase, the percentage of each transmission subtask in the application task in the total resources of the task is calculated to complete the traffic level resource allocation;

[0010] According to the task-level resource allocation strategy and the flow-level resource allocation strategy, the resources of the simulation tasks are dynamically adjusted and scheduled according to the task priority.

[0011] A second aspect of an embodiment of the present invention provides a multi-task collaborative traffic scheduling system based on a port bandwidth reservation mode, the system comprising:

[0012] The distributed computing task packet issuing component is used to obtain the collaborative traffic scheduling simulation task parameters and the simulation task service quality requirements, and issue task-level resource allocation applications and traffic-level resource allocation applications; when the task network bandwidth resources change, the task network bandwidth resource change notification message is sent to the distributed computing task packet receiving component;

[0013] The distributed computing task packet receiving component is used to receive the collaborative traffic sent by the distributed computing task packet sending component and complete message parsing and statistics; after any collaborative traffic transmission is completed, it feeds back a collaborative traffic transmission end confirmation message to the listening port of the distributed computing task packet sending component; in response to the task network bandwidth resource change notification message, it constructs and sends a resource change notification confirmation message to the listening port of the distributed computing task packet sending component;

[0014] The network scheduling component responds to the task-level resource allocation application issued by the distributed computing task package component to allocate network bandwidth resources and formulate a task-level resource allocation strategy; responds to the flow-level resource allocation application issued by the distributed computing task package component to generate the resource ratio of each transmission stage and formulate a flow-level resource allocation strategy; and dynamically adjusts the resources of the task according to the task priority;

[0015] An SDN switch, the SDN switch includes a control plane and a data plane; the control plane responds to the flow-level resource allocation strategy issued by the network scheduling component and generates a flow table, and responds to the network bandwidth resource query request of the network scheduling component; the data plane receives the flow table issued by the control plane, reserves bandwidth resources for different tasks according to the flow table, and forwards the task traffic to the next-hop SDN switch or the distributed computing task packet receiving component.

[0016] A third aspect of an embodiment of the present invention provides an electronic device, comprising one or more processors, for implementing the above-mentioned multi-task collaborative traffic scheduling method based on the port bandwidth reservation mode.

[0017] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-mentioned multi-task collaborative traffic scheduling method based on the port bandwidth reservation mode.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] In the scenario of multiple computing tasks, the present invention designs a two-level scheduling mechanism (task-level resource allocation, flow-level resource allocation) based on bandwidth resource reservation for multiple distributed computing tasks based on the characteristics of the computing stage and the transmission stage of distributed computing tasks, so as to optimize the overall task time and the single task time, improve the utilization rate of the cluster network, and guide the resource allocation algorithm to achieve the goal of meeting more task requirements. At the same time, the present invention will allow high-priority tasks to preemptively call resources, adjust resources based on the use of the resource pool, the characteristics of each task and the execution status, and design a resource change notification mechanism and notification protocol to ensure the packet loss rate of task traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0021] Figure 1 It is a schematic diagram of the architecture of a multi-task collaborative traffic scheduling system based on a port bandwidth reservation mode provided by an embodiment of the present invention;

[0022] Figure 2 It is a schematic diagram of a service message format and a resource change notification message provided by an embodiment of the present invention;

[0023] Figure 3 It is a schematic diagram of the process of total bandwidth resource change notification provided by an embodiment of the present invention;

[0024] Figure 4 It is a business flow chart of a package issuing component provided by an embodiment of the present invention;

[0025] Figure 5 is a schematic diagram of a network scheduling component provided by an embodiment of the present invention;

[0026] Figure 6 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0028] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0029] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0030] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0031] like Figure 1 As shown, an embodiment of the present invention provides a multi-task collaborative traffic scheduling method based on a port bandwidth reservation mode, the method comprising:

[0032] Step S1, obtaining collaborative traffic scheduling simulation task parameters and simulation task service quality requirements.

[0033] Furthermore, the collaborative traffic scheduling simulation task parameters include: the unique identifier of the distributed computing task job_id, the statistical characteristics of each computing stage in the distributed computing task (including data processing efficiency ω, random parameter θ), the statistical characteristics of each transmission stage of the distributed computing task (i.e., the size of the collaborative traffic), and the directed acyclic graph of the distributed computing task that characterizes the dependency relationship between each computing stage and the transmission stage (in this example, the directed acyclic graph uses the adjacency matrix M dag express).

[0034] Furthermore, the simulation task service quality requirements include: the priority of the distributed computing task job_id , the latest completion time deadline of the distributed computing task job_id .

[0035] Step S2, when the collaborative traffic scheduling simulation task is started, the task bandwidth resource application is performed according to the simulation task parameters, the network bandwidth resource pool of the collaborative traffic scheduling simulation task is obtained, and the task-level resource allocation is completed.

[0036] Specifically, when the collaborative traffic scheduling simulation task is started, the unique identifier of the simulation task, job_id, and the task priority priority job_id , dependency matrix Time for each calculation stage Remaining transmission capacity in each transmission stage It is encapsulated into a task bandwidth resource request to apply for the total network bandwidth resources of the collaborative traffic scheduling simulation task, that is, the network bandwidth resource pool of the collaborative traffic scheduling simulation task.

[0037] Step S3, when the collaborative traffic scheduling simulation task is started or any collaborative traffic transmission subtask is completed, according to the dependency relationship between the calculation phase and the transmission phase in the simulation task and the corresponding characteristics of the transmission phase, the percentage of each transmission subtask in the application task in the total resources of the task is calculated to complete the traffic-level resource allocation.

[0038] Specifically, when the collaborative traffic scheduling simulation task is started or any collaborative traffic transmission subtask is completed, the dependency matrix of the simulation task is And the feature vector with edge as dimension Encapsulated into a request, to apply for the percentage of each transmission subtask in the total task resources, to complete the flow level resource allocation. Including the remaining transmission amount during the transmission phase The level of the starting endpoint in the transmission phase in the DAG, the remaining computing time of the starting endpoint in the transmission phase, and the in-degree and out-degree of the starting endpoint.

[0039] Step S4: Dynamically adjust and schedule the resources of the simulation task according to the task-level resource allocation strategy and the flow-level resource allocation strategy, and according to the task priority.

[0040] The method further comprises:

[0041] When the total task bandwidth corresponding to the calculation phase in the simulation task and / or the bandwidth proportion corresponding to the transmission task in the simulation task is adjusted, in response to the resource change notification message ( Figure 2 ),like Figure 3 As shown, a new thread is started, such as computing node k, according to the total value of the in-degree flow Total outflow Data processing efficiency ω and random parameter θ, get its calculation time Blocking Threads After completing the simulation of the calculation phase, recreate the transmission process for the transmission phase of the simulation task, obtain the resource allocation ratio of each transmission phase, and encapsulate and transmit the message.

[0042] It should be noted that the calculation phase in the simulation task refers to the completion of all pre-transmission tasks of the calculation phase, that is, the execution starts after the in-degree traffic transmission is completed.

[0043] The embodiment of the present invention proposes a multi-task collaborative traffic scheduling system based on a port bandwidth reservation mode, the system comprising:

[0044] Distributed computing task dispatching components, such as Figure 4 As shown, it is used to obtain the collaborative traffic scheduling simulation task parameters and the simulation task service quality requirements, and issue task-level resource allocation applications and traffic-level resource allocation applications; when the task network bandwidth resources change, send the task network bandwidth resource change notification message to the distributed computing task packet receiving component.

[0045] The collaborative traffic scheduling simulation task parameters include: a distributed computing task unique identifier job_id, statistical characteristics of each computing stage in the distributed computing task, statistical characteristics of each transmission stage in the distributed computing task, and dependencies used to characterize each computing stage and each transmission stage in the distributed computing task;

[0046] The statistical characteristics of each computing stage in the distributed computing task include data processing efficiency ω and random parameter θ;

[0047] The statistical characteristics of each transmission phase in the distributed computing task include the size of the collaborative traffic;

[0048] The simulation task service quality requirements include: the priority of the distributed computing task job_id , the latest completion time deadline of the distributed computing task job_id .

[0049] Furthermore, if Figure 2 As shown, the service message includes: Coflow Payload, Coflow Header, UDP Header and IP Header, wherein the Coflow Header includes: job id, pkt_flag, coflow_from, coflow_to, pkt_num, monitor_port, rate_limit, job_finish and cf_finish.

[0050] Furthermore, if Figure 2As shown, the task network bandwidth resource change notification message includes: CoflowPayload, Coflow Header, UDP Header and IP Header, wherein the Coflow Header includes: job id, pkt_flag, adjust_ack, adjust_ack and monitor_port.

[0051] The distributed computing task packet receiving component is used to receive the collaborative traffic sent by the distributed computing task packet sending component and complete the parsing and statistics of the message; after any collaborative traffic transmission is completed, it feeds back the collaborative traffic transmission end confirmation message to the listening port of the distributed computing task packet sending component; in response to the task network bandwidth resource change notification message, it constructs and sends the resource change notification confirmation message to the listening port of the distributed computing task packet sending component.

[0052] It should be noted that when the task network bandwidth resources change, the distributed computing task packaging component needs to notify and confirm with the distributed computing task packaging component. The specific process includes: the distributed computing task packaging component sends a task network bandwidth resource change notification message, the distributed computing task packaging component receives the task network bandwidth resource change notification message and replies with a confirmation message. After completing the notification and confirmation, the distributed computing task packaging component will send business messages according to the updated task network bandwidth resource ratio.

[0053] Furthermore, when the task network bandwidth resources change, that is, when the total task bandwidth corresponding to the calculation stage in the simulation task and / or the bandwidth proportion corresponding to the transmission task in the simulation task is adjusted, in response to the resource change notification message, a new thread corresponding to the calculation stage is started, and the calculation node is recorded as k. Total outflow Data processing efficiency ω and random parameter θ, get its calculation time Blocking Threads After the calculation phase is completed, the transmission process is recreated for the transmission phase of the simulation task, and the resource allocation ratio of each transmission phase is obtained.

[0054] Network scheduling components, such as Figure 5 As shown, in response to the task-level resource allocation application issued by the distributed computing task packaging component to allocate network bandwidth resources, a task-level resource allocation strategy is formulated; in response to the flow-level resource allocation application issued by the distributed computing task packaging component to generate the resource ratio of each transmission stage, a flow-level resource allocation strategy is formulated; and the resources of the task are dynamically adjusted according to the task priority.

[0055] Furthermore, when the collaborative traffic scheduling simulation task is started, the unique identifier of the simulation task, job_id, and the task priority priority job_id , the dependency matrix used to characterize each computing stage and each transmission stage in distributed computing tasks Time for each calculation stage Remaining transmission capacity in each transmission stage It is encapsulated into a task bandwidth resource request to apply for the total network bandwidth resources of the collaborative traffic scheduling simulation task, obtain the network bandwidth resource pool of the collaborative traffic scheduling simulation task, and complete the task-level resource allocation.

[0056] Furthermore, when the collaborative traffic scheduling simulation task is started or any collaborative traffic transmission subtask is completed, the dependencies between the computing phases and the transmission phases in the distributed computing task are represented. And the corresponding characteristics of the transmission stage Encapsulated into a request, to apply for the percentage of each transmission subtask of the task in the total task resources, to complete the flow level resource allocation; Among them, the corresponding characteristics of the transmission stage Including the remaining transmission amount during the transmission phase The level of the starting endpoint of the transmission phase in the directed acyclic graph, the remaining computing time of the starting endpoint of the transmission phase, and the in-degree and out-degree of the starting endpoint of the transmission phase; the directed acyclic graph is used to characterize the dependency relationship between each computing phase and each transmission phase in the distributed computing task.

[0057] An SDN switch, the SDN switch includes a control plane and a data plane; the control plane responds to the flow-level resource allocation strategy issued by the network scheduling component and generates a flow table, and responds to the network bandwidth resource query request of the network scheduling component; the data plane receives the flow table issued by the control plane, reserves bandwidth resources for different tasks according to the flow table, and forwards the task traffic to the next-hop SDN switch or the distributed computing task packet receiving component.

[0058] Furthermore, the data plane parses the received business message and forwards the task traffic according to the bandwidth resources set by the network scheduling component based on the unique identifier job_id of the distributed computing task.

[0059] Furthermore, the network scheduling component continuously monitors requests for simulation tasks, formulates two-level strategies of task-level resource allocation strategy and traffic-level resource allocation strategy for each simulation task, interacts with the distributed computing task packaging component to complete the status acquisition and policy formulation of the simulation task, and interacts with the SDN switch to complete the acquisition of network environment status and policy issuance.

[0060] The present specification also provides a computer-readable storage medium, which stores a computer program, and the computer program can be used to execute the above-mentioned data synchronization method.

[0061] This manual also provides Figure 6 The schematic structure diagram of the electronic device shown in FIG. Figure 6 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above data synchronization method.

[0062] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0063] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0064] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0065] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0066] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0067] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0068] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0069] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0071] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0072] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0073] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0074] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0075] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0077] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0078] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A multi-task collaborative traffic scheduling method based on port bandwidth reservation mode, characterized in that: The method comprises: Obtain collaborative traffic scheduling simulation task parameters and simulation task service quality requirements; When the collaborative traffic scheduling simulation task is started, the task bandwidth resource application is performed according to the simulation task parameters and the simulation task service quality requirements, the network bandwidth resource pool of the collaborative traffic scheduling simulation task is obtained, and the task-level resource allocation is completed; including: When the collaborative traffic scheduling simulation task is started, the unique identifier of the simulation task, job_id, task priority , the dependency matrix used to characterize each computing stage and each transmission stage in distributed computing tasks , each calculation stage time , the remaining transmission amount in each transmission stage Encapsulate into task bandwidth resource request to apply for the total network bandwidth resource of the collaborative traffic scheduling simulation task, obtain the network bandwidth resource pool of the collaborative traffic scheduling simulation task, and complete task-level resource allocation; When the collaborative traffic scheduling simulation task is started or any collaborative traffic transmission subtask is completed, according to the dependency between the calculation phase and the transmission phase in the collaborative traffic scheduling simulation task and the corresponding characteristics of the transmission phase, the percentage of the total resources of the collaborative traffic scheduling simulation task occupied by each transmission subtask in the collaborative traffic scheduling simulation task is applied to complete the traffic level resource allocation; including: When the collaborative traffic scheduling simulation task is started or any collaborative traffic transmission subtask is completed, the dependencies between the computing phases and the transmission phases in the distributed computing task are represented. , and the corresponding characteristics of the transmission phase Encapsulate into a request to apply for the percentage of each transmission subtask in the collaborative traffic scheduling simulation task to the total resources of the collaborative traffic scheduling simulation task, and complete the traffic level resource allocation; Among them, the characteristics corresponding to the transmission stage Including the remaining transmission amount during the transmission phase , the level of the starting endpoint of the transmission phase in the directed acyclic graph, the remaining computing time of the starting endpoint of the transmission phase, and the in-degree and out-degree of the starting endpoint of the transmission phase; the directed acyclic graph is used to characterize the dependency between each computing phase and each transmission phase in the distributed computing task; According to the task-level resource allocation strategy and the flow-level resource allocation strategy, the resources of the simulation tasks are dynamically adjusted and scheduled according to the task priority.

2. The multi-task collaborative traffic scheduling method based on port bandwidth reservation mode according to claim 1 is characterized in that: The collaborative traffic scheduling simulation task parameters include: a distributed computing task unique identifier job_id, statistical characteristics of each computing stage in the distributed computing task, statistical characteristics of each transmission stage in the distributed computing task, and dependencies used to characterize each computing stage and each transmission stage in the distributed computing task; The statistical characteristics of each computing stage in the distributed computing task include data processing efficiency ω and random parameter θ; The statistical characteristics of each transmission phase in the distributed computing task include the size of the collaborative traffic; The simulation task service quality requirements include: the priority of distributed computing tasks , the latest completion time of distributed computing tasks .

3. The multi-task collaborative traffic scheduling method based on port bandwidth reservation mode according to claim 1 is characterized in that: The method further comprises: When the total bandwidth of the task corresponding to the calculation stage in the simulation task and / or the bandwidth ratio corresponding to the transmission task in the simulation task is adjusted, in response to the resource change notification message, a new thread corresponding to the calculation stage is started, and the calculation node is recorded as k. , total outflow , data processing efficiency ω and random parameter θ, and the calculation time is obtained , blocking the thread After the calculation phase is completed, the transmission process is recreated for the transmission phase of the simulation task, and the resource allocation ratio of each transmission phase is obtained.

4. A multi-task collaborative traffic scheduling system based on port bandwidth reservation mode, characterized in that: The system comprises: The distributed computing task packet issuing component is used to obtain the collaborative traffic scheduling simulation task parameters and the simulation task service quality requirements, and issue task-level resource allocation applications and traffic-level resource allocation applications; when the task network bandwidth resources change, the task network bandwidth resource change notification message is sent to the distributed computing task packet receiving component; The distributed computing task packet receiving component is used to receive the collaborative traffic sent by the distributed computing task packet sending component and complete message parsing and statistics; after any collaborative traffic transmission is completed, it feeds back a collaborative traffic transmission end confirmation message to the listening port of the distributed computing task packet sending component; in response to the task network bandwidth resource change notification message, it constructs and sends a resource change notification confirmation message to the listening port of the distributed computing task packet sending component; The network scheduling component responds to the task-level resource allocation application issued by the distributed computing task package component to allocate network bandwidth resources and formulate a task-level resource allocation strategy; responds to the flow-level resource allocation application issued by the distributed computing task package component to generate the resource ratio of each transmission subtask and formulate a flow-level resource allocation strategy; and dynamically adjusts the resources of the task according to the task priority; When the collaborative traffic scheduling simulation task is started, the unique identifier of the simulation task, job_id, task priority , the dependency matrix used to characterize each computing stage and each transmission stage in distributed computing tasks , each calculation stage time , the remaining transmission amount in each transmission stage Encapsulate into task bandwidth resource request to apply for the total network bandwidth resource of the collaborative traffic scheduling simulation task, obtain the network bandwidth resource pool of the collaborative traffic scheduling simulation task, and complete task-level resource allocation; When the collaborative traffic scheduling simulation task is started or any collaborative traffic transmission subtask is completed, the dependencies between the computing phases and the transmission phases in the distributed computing task are represented. , and the corresponding characteristics of the transmission phase Encapsulated into a request, to apply for the percentage of each transmission subtask of the task in the total task resources, to complete the flow level resource allocation; Among them, the corresponding characteristics of the transmission stage Including the remaining transmission amount during the transmission phase , the level of the starting endpoint of the transmission phase in the directed acyclic graph, the remaining computing time of the starting endpoint of the transmission phase, and the in-degree and out-degree of the starting endpoint of the transmission phase; the directed acyclic graph is used to characterize the dependency between each computing phase and each transmission phase in the distributed computing task; An SDN switch, the SDN switch includes a control plane and a data plane; the control plane responds to the flow-level resource allocation strategy issued by the network scheduling component and generates a flow table, and responds to the network bandwidth resource query request of the network scheduling component; the data plane receives the flow table issued by the control plane, reserves bandwidth resources for different tasks according to the flow table, and forwards the task traffic to the next-hop SDN switch or the distributed computing task packet receiving component.

5. A multi-task collaborative traffic scheduling system based on port bandwidth reservation mode according to claim 4, characterized in that: When the task network bandwidth resource changes, sending a task network bandwidth resource change notification message to the distributed computing task packet receiving component; the distributed computing task packet receiving component responds to the task network bandwidth resource change notification message, constructs and sends a resource change notification confirmation message to the listening port of the distributed computing task packet receiving component, including: When the total bandwidth of the task corresponding to the calculation stage in the simulation task and / or the bandwidth ratio corresponding to the transmission task in the simulation task is adjusted, in response to the resource change notification message, a new thread corresponding to the calculation stage is started, and the calculation node is recorded as k. , total outflow , data processing efficiency ω and random parameter θ, and the calculation time is obtained , blocking the thread After the calculation phase is completed, the transmission process is recreated for the transmission phase of the simulation task, and the resource allocation ratio of each transmission subtask is obtained.

6. An electronic device, characterized in that: It includes one or more processors, used to implement the multi-task collaborative traffic scheduling method based on the port bandwidth reservation mode according to any one of claims 1-3.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement the multi-task collaborative traffic scheduling method based on the port bandwidth reservation mode according to any one of claims 1 to 3.

Citation Information

Patent Citations

  • Reserved resource configuration method and device, equipment and storage medium

    CN113938442A

  • Task processing method and device and computing equipment

    CN114116189A

  • Service bearer distribution method and system based on 5G power

    CN114125904A

  • Cluster calculation flow simulation method, system and device based on packet receiving and sending cooperation

    CN114465941A

  • Joint Network and Task Scheduling

    US20170286180A1