A unified communication agent supporting concurrent flow scheduling in cluster computing

By deploying unified communication agents in cluster computing, merging and splitting flows to reduce the number of concurrent flows, the network performance problems caused by concurrent flows in cluster computing are solved, and the network performance and coordination are achieved. Flow scheduling efficiency.

CN115633089BActive Publication Date: 2024-12-03JIANGSU FUTURE NETWORKS INNOVATION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211083920.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-06
Publication Date
2024-12-03
Estimated Expiration
2042-09-06

AI Technical Summary

Technical Problem

In cluster computing, a large number of concurrent flows cause switch queues to grow longer, affecting network performance, resulting in increased latency and packet loss. The existing co-stream scheduling algorithms have failed to effectively solve this problem.

Method used

By deploying a unified communication agent on machines participating in cluster computing, the number of flows between servers is reduced, the communication needs are forwarded to the proxy and interact with the co-stream scheduling algorithm, the rate or sequence of the unified stream is calculated, and data transmission, merging and segmentation are carried out. Stream to reduce the number of concurrent streams.

Benefits of technology

It effectively reduces network latency and packet loss rate, improves network performance, and is well coupled with existing co-streaming scheduling algorithms, reduces co-streaming completion time, and adapts to the complex topology of data centers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115633089B_ABST
    Figure CN115633089B_ABST
Patent Text Reader

Abstract

The present invention provides a unified communication agent that supports concurrent flow scheduling in cluster computing. By deploying the unified communication agent on machines participating in cluster computing, the number of flows between two pairs of servers is reduced to one. Specific invention highlights include: ① When a cluster computing framework on a server has a communication demand, the framework forwards the communication demand to the agent of the present invention deployed on the machine, rather than directly initiating a data flow; ② After collecting the communication demand, it interacts with the currently deployed co-flow scheduling algorithm to calculate the rate or order of the unified flow; ③ Initiate a data flow based on the result calculated in ②, and distribute the transmitted data to the cluster computing framework on the server respectively; the present invention can be applied to a data center that deploys a cluster computing framework; by reducing the number of concurrent flows, the network performance is optimized to accelerate the completion speed of cluster computing tasks and improve computing efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of cluster computing, in particular to a unified communication agent supporting concurrent flow scheduling in cluster computing. Background Art

[0002] The process of mainstream cluster computing frameworks is highly structured. Let's take a computing task in Hadoop as an example. The data is first divided into m parts and processed by m mappers deployed on some machines; then it is processed by r reducers deployed on some machines, and finally the results are summarized to get the final calculation result. The calculation of each reducer requires the output data of each mapper as input data. Figure 1 As shown in the figure, during the data transmission phase of the above tasks, each cluster computing task will have m×r concurrent data flows. If n cluster computing tasks are run at the same time, the number of concurrent flows in the cluster can reach up to n×m×r.

[0003] Co-flow was proposed to optimize the completion time of cluster computing tasks. Based on the abstraction of co-flow, many co-flow scheduling algorithms were proposed. These co-flow scheduling algorithms can be divided into two categories, one is rate-based scheduling algorithm, and the other is order-based scheduling algorithm. Taking Varys and Aalo as examples, the rate-based scheduling algorithm first calculates the priority relationship between co-flows, and then allocates the sending rate to each flow in the co-flow according to these relationships. Taking Sincronia as an example, the order-based scheduling algorithm also uses some planning algorithms or heuristic algorithms to determine the priority relationship for co-flows, but does not allocate a rate for each flow. The order-based scheduling algorithm believes that as long as the completion order between co-flows is guaranteed (that is, high-priority co-flows are completed before low-priority ones), good scheduling can be achieved. Therefore, the order-based scheduling algorithm often uses some underlying mechanisms (such as the underlying network protocol stack with DSCP enabled) to ensure the completion order of co-flows.

[0004] In data centers, many congestion control algorithms have been proposed. These algorithms are essential to achieve the service goals of high throughput and low latency for data flows. DCTCP is one of the most widely deployed data center congestion control algorithms, which aims to reduce the queue length on the switch to improve data flow performance.

[0005] Data center congestion control algorithms such as DCTCP cannot maintain an appropriate queue length to avoid packet loss when there are too many concurrent flows. In steady state, the maximum queue length q in a network using DCTCP is max The relationship between the ECN threshold k and the number of concurrent flows n is as follows:

[0006] q max =k+n

[0007] When the number of concurrent flows in a data center increases, the maximum queue length in a network using DCTCP will gradually increase, and even packet loss may occur, which will seriously affect network performance.

[0008] According to the survey, almost all the current co-flow scheduling algorithms ignore the impact that too many concurrent flows may have on network performance.

[0009] Hadoop has a workaround for the concurrent stream problem: limit the number of streams that each reducer can initiate simultaneously. Figure 2 As shown in the figure, a simple simulation experiment illustrates the problem of this method: the number of streams initiated by each reducer is difficult to set. Setting it too large or too small will lead to an increase in CCT. Summary of the invention

[0010] Considering that a large number of concurrent flows in cluster computing will cause the queue on the switch to become longer, on the one hand, it will affect the delay of delay-sensitive flows, and on the other hand, it may cause packet loss and affect CCT; the present invention aims to reduce the number of concurrent flows in cluster computing, so as to achieve the purpose of improving network performance.

[0011] To achieve the above object, the present invention provides the following technical solutions:

[0012] A unified communication agent supporting concurrent flow scheduling in cluster computing reduces the number of flows between two pairs of servers to one by deploying the unified communication agent on machines participating in cluster computing, specifically comprising:

[0013] ① When the cluster computing framework on the server has a communication demand, the framework forwards the communication demand to the agent of the present invention deployed on the machine instead of directly initiating a data flow;

[0014] ②After collecting the communication requirements, it will interact with the currently deployed co-flow scheduling algorithm to calculate the rate or order of the unified flow;

[0015] ③ Initiate data flow based on the results calculated in ②, and distribute the transmitted data to the cluster computing framework on the server respectively.

[0016] The product form of the present invention is a unified communication agent software (401), which supports the function of receiving the communication needs of the cluster computing framework and transmitting data on its behalf. The unified communication agent software (401) mainly includes two parts: a communication demand processing module (402) and a data transmission module (403); the communication demand processing module (402) will pass the communication demand to the data transmission module (403), and the data transmission module (403) will uniformly initiate data flow to transmit data according to the communication demand.

[0017] The core method flow is:

[0018] S502, receiving communication requirements of a cluster computing framework;

[0019] S503: For all communication requirements, run the co-flow scheduling algorithm to calculate their appropriate transmission rates or priorities;

[0020] S504, calculating the transmission rate or priority of the unified flow to each other server according to the result of the co-flow scheduling algorithm;

[0021] S505, transmitting data through a unified stream;

[0022] Among them, step S502, step S503 and step S504 are mainly completed through the control module (601), and step S505 is mainly completed through the data module (606). The control module (601) is equivalent to the demand processing module (402) in the unified communication agent software (401), and the data module (606) is equivalent to the data transmission module (403) in the unified communication agent software (401).

[0023] The step S502 specifically includes: receiving the communication demand of the cluster computing framework through the communication demand receiving module (602).

[0024] The step S503 specifically includes: running the existing co-flow scheduling algorithm through the co-flow scheduling algorithm module (603), taking the communication requirements of the cluster computing framework received in step S502 as input, and outputting a scheduled data flow list L, wherein each flow is defined as a four-tuple <address pair, size, rate, priority>; the rate field is only valid when the deployed co-flow scheduling algorithm is a rate-based scheduling algorithm, and the same applies to the priority field.

[0025] The step S504 specifically includes: taking the data stream list L outputted in step S503 as input through the unified stream merging module (604), merging the scheduled streams in the data stream list L into a unified stream, and outputting the unified stream list L. uni ; Then, through the unified stream selection and segmentation module (605), the unified stream list L uni The unified stream in is selected and split according to the maximum number of streams that can be sent, and finally the list L of the actually sent streams is output. act .

[0026] The unified stream merging module (604) merges the scheduled streams into a unified stream, specifically including the following contents:

[0027] S702. Receive the data stream list L output by the scheduling algorithm module (603) as input;

[0028] S703. Determine whether there are any unmerged scheduling flows in L;

[0029] S704. If there are still unmerged scheduled flows in L, take one of them as the flow to be merged;

[0030] S705. Determine the unified flow list L uni Whether there is a unified flow corresponding to the flow taken out in S704 (with the same source and destination IP address pair);

[0031] S706. If there is no corresponding unified flow, create an empty unified flow (the source and destination IP address pair is the same as the extracted flow, and the size, rate, and priority are all 0) and put it into the unified flow list L uni ;

[0032] S707. Determine whether the scheduling algorithm is rate-based;

[0033] S708. If the scheduling algorithm is rate-based, then the rate-based method is used to merge the extracted flow and the unified flow (the size and rate of the flow are added together), and then return to S703;

[0034] S709. If the scheduling algorithm is not rate-based, it is determined to be sequence-based, and the two flows are merged using a sequence-based method (the flow with the highest priority is taken as the unified flow. If the two flows have the same priority, the sizes are added together and the priority remains unchanged), and then return to S703;

[0035] S710. If there is no unmerged scheduled flow in L, that is, all flows have been merged, then output the unified flow list L uni .

[0036] The unified stream selection and segmentation module (605) specifically includes the following contents in the unified stream selection and segmentation:

[0037] S802. Receive the output L of the unified stream merging module (604) uni As input;

[0038] S803. Determine L act Whether the number of midstreams has reached the upper limit;

[0039] S804. If L act If the number of midstreams has not reached the upper limit, then L uni Are there any other items not included in L act The unified flow of

[0040] S805. If L uni If there are still flows not included in L uni Take out a flow with the highest rate or priority and add it to L act , return to S803;

[0041] S806. If L uni All have been taken out, then from L act Select a flow with the highest rate or priority;

[0042] S807. Determine whether the scheduling algorithm is rate-based;

[0043] S808. If the scheduling algorithm is rate-based, then the rate-based method is used to split the selected stream into two (the size and rate of the selected stream are equally divided into two new streams), and then return to S803;

[0044] S809. If the scheduling algorithm is not rate-based, it is determined to be sequence-based, and the selected stream is split into two using a sequence-based method (the size of the selected stream is equally divided into two new streams, and the priority of the two new streams remains the same as the selected stream), and then returns to S803;

[0045] S810. If L act The number of flows reaches the upper limit, the algorithm ends, and the list of flows actually emitted is output. act .

[0046] The step S505 specifically includes: transmitting the data of the cluster computing task according to the information calculated by the control module (601) through the data module (606), which is specifically completed by the cluster computing data interaction module (607) and the data transmission module (608), wherein the cluster computing data interaction module (607) uses the communication demand information transmitted by the communication demand receiving module (602) to collect the data that the mapper (sender) needs to transmit or distributes the network data received by the data transmission module (608) to the reducer (receiver), the mapper's communication information is usually the storage location of the data to be transmitted, and the reducer's information is usually the sender's IP address and port number; the data transmission module (608) uses the unified flow queue actually sent finally given by the unified flow selection and segmentation module (605) to send the unified flow transmission data according to the information of this queue.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention discloses a network flow proxy method in cluster computing. By proxying the communication demands in cluster computing, the communication demands are integrated according to source-destination address pairs, thereby reducing the number of flows in cluster computing and improving network performance. By maintaining the original scheduling attributes when merging flows, the method can be well coupled with the existing co-flow scheduling algorithm and reduce the co-flow completion time. In consideration of the complexity of the actual topology of a data center, a max-min fairness mechanism is used to adapt to the topology of the data center to the maximum extent while maintaining the scheduling order when the number of flows allowed to be initiated is limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the number of data flows for cluster calculation;

[0050] Figure 2 It is a simulation experiment of Hadoop under DCTCP;

[0051] Figure 3 It is a schematic diagram of the principle of the present invention;

[0052] Figure 4 This is a schematic diagram of the core product of the present invention;

[0053] Figure 5 This is a flow chart of the core method of the present invention;

[0054] Figure 6 This is a module block diagram of Embodiment 1 of the present invention;

[0055] Figure 7 This is a flow chart of Embodiment 1 of the present invention (unified flow merging);

[0056] Figure 8 This is a flow chart of Embodiment 1 of the present invention (unified flow selection and segmentation);

[0057] Fig. 9 This is the effect of the first embodiment of the present invention when each reducer initiates a different number of streams;

[0058] Fig.10 This is the effect of the first embodiment of the present invention under multipath topology;

[0059] Fig.11 This is the effect of the first embodiment of the present invention under a large topology;

[0060] Fig.12 This is the effect of the first embodiment of the present invention under different workloads. DETAILED DESCRIPTION

[0061] In order to clarify the technical problems, technical solutions, implementation processes and performance demonstrations, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are for explanation only. The present invention is not intended to limit the present invention. Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0062] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0063] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0064] Example 1

[0065] like Figure 3 As shown, the present invention is a method for reducing the number of data flows in cluster computing by using a unified communication agent. By deploying a unified communication agent on the machines participating in cluster computing, the number of flows between two pairs of servers is reduced to one. By this method, the number of flows in cluster computing is greatly reduced, ensuring normal network performance; specific invention points include:

[0066] (1) When a communication demand occurs in the cluster computing framework on the server, the framework forwards the communication demand to the agent of the present invention deployed on the machine instead of directly initiating a data flow.

[0067] (2) After collecting the communication demand, the present invention will interact with the currently deployed co-flow scheduling algorithm to calculate the rate (if the deployed co-flow scheduling algorithm is based on rate) or order (if the deployed co-flow scheduling algorithm is based on order) of the unified flow.

[0068] (3) Finally, the present invention initiates a data flow based on the result of the calculation in (2), and distributes the transmitted data to the cluster computing framework on the server respectively.

[0069] The present invention can be applied to data centers that deploy cluster computing frameworks. By reducing the number of concurrent flows, the network performance is optimized to accelerate the completion of cluster computing tasks and improve computing efficiency, specifically:

[0070] A unified communication agent supporting concurrent flow scheduling in cluster computing reduces the number of flows between two pairs of servers to one by deploying the unified communication agent on machines participating in cluster computing, specifically comprising:

[0071] ① When the cluster computing framework on the server has a communication demand, the framework forwards the communication demand to the agent of the present invention deployed on the machine instead of directly initiating a data flow;

[0072] ②After collecting the communication requirements, it will interact with the currently deployed co-flow scheduling algorithm to calculate the rate or order of the unified flow;

[0073] ③ Initiate data flow based on the results calculated in ②, and distribute the transmitted data to the cluster computing framework on the server respectively.

[0074] like Figure 4 As shown, the product form of the present invention is a unified communication agent software (401), which supports the function of receiving the communication needs of the cluster computing framework and transmitting data on its behalf. The unified communication agent software (401) mainly includes two parts: a communication demand processing module (402) and a data transmission module (403); the communication demand processing module (402) will pass the communication demand to the data transmission module (403), and the data transmission module (403) will uniformly initiate data flow to transmit data according to the communication demand.

[0075] like Figure 5 As shown, the core method flow is:

[0076] S502, receiving communication requirements of a cluster computing framework;

[0077] S503: For all communication requirements, run the co-flow scheduling algorithm to calculate their appropriate transmission rates or priorities;

[0078] S504, calculating the transmission rate or priority of the unified flow to each other server according to the result of the co-flow scheduling algorithm;

[0079] S505, transmitting data through a unified stream;

[0080] This embodiment restores the window in the form of a signaling message generated by the switch;

[0081] like Figure 6 As shown, the main modules within a unified communication agent software and the connections and interactions between the modules are demonstrated.

[0082] Among them, step S502, step S503 and step S504 are mainly completed through the control module (601), and step S505 is mainly completed through the data module (606); the control module (601) is equivalent to the demand processing module (402) in the unified communication agent software (401), and the data module (606) is equivalent to the data transmission module (403) in the unified communication agent software (401).

[0083] The step S502 specifically includes: receiving the communication demand of the cluster computing framework through the communication demand receiving module (602).

[0084] The step S503 specifically includes: running the existing co-flow scheduling algorithm through the co-flow scheduling algorithm module (603), taking the communication requirements of the cluster computing framework received in step S502 as input, and outputting a scheduled data flow list L, wherein each flow is defined as a four-tuple <address pair, size, rate, priority>; the rate field is only valid when the deployed co-flow scheduling algorithm is a rate-based scheduling algorithm, and the same applies to the priority field.

[0085] The step S504 specifically includes: using the unified flow merging module (604) to take the data flow list L outputted in step S503 as input, merging the scheduled flows in the data flow list L into a unified flow, and outputting the unified flow list Luni; then using the unified flow selection and segmentation module (605), the unified flows in the unified flow list Luni are selected and segmented according to the maximum number of flows that can be sent, and finally outputting the list Lact of flows actually sent.

[0086] The step S505 specifically includes: transmitting the data of the cluster computing task according to the information calculated by the control module (601) through the data module (606), which is specifically completed by the cluster computing data interaction module (607) and the data transmission module (608), wherein the cluster computing data interaction module (607) uses the communication demand information transmitted by the communication demand receiving module (602) to collect the data that the mapper (sender) needs to transmit or distributes the network data received by the data transmission module (608) to the reducer (receiver), the mapper's communication information is usually the storage location of the data to be transmitted, and the reducer's information is usually the sender's IP address and port number; the data transmission module (608) uses the unified flow queue actually sent finally given by the unified flow selection and segmentation module (605) to send the unified flow transmission data according to the information of this queue.

[0087] The functions of each module in this embodiment are described in detail as follows:

[0088] 601. Control module of unified communication agent software;

[0089] 602. A communication demand receiving module, used to receive communication demands of a cluster computing framework;

[0090] 603. Co-flow scheduling algorithm module, where the existing co-flow scheduling algorithm is run, the communication requirements collected in 602 are accepted as input, and a list of scheduled data flows L is output, where each flow is defined as a four-tuple <address pair, size, rate, priority> (the rate field is only valid when the deployed co-flow scheduling algorithm is a rate-based scheduling algorithm, and the same applies to the priority field);

[0091] 604. Unified flow merging module, in this module, the scheduled flows in L are merged into a unified flow. Figure 7 As shown, it shows the logic of merging the scheduling flows into a unified flow;

[0092] The unified stream merging module (604) merges the scheduled streams into a unified stream, specifically including the following contents:

[0093] S702. Receive the data stream list L output by the scheduling algorithm module (603) as input;

[0094] S703. Determine whether there are any unmerged scheduling flows in L;

[0095] S704. If there are still unmerged scheduled flows in L, take one of them as the flow to be merged;

[0096] S705. Determine the unified flow list L uni Whether there is a unified flow corresponding to the flow taken out in S704 (with the same source and destination IP address pair);

[0097] S706. If there is no corresponding unified flow, create an empty unified flow (the source and destination IP address pair is the same as the extracted flow, and the size, rate, and priority are all 0) and put it into the unified flow list L uni ;

[0098] S707. Determine whether the scheduling algorithm is rate-based;

[0099] S708. If the scheduling algorithm is rate-based, then the rate-based method is used to merge the extracted flow and the unified flow (the size and rate of the flow are added together), and then return to S703;

[0100] S709. If the scheduling algorithm is not rate-based, it is determined to be sequence-based, and the two flows are merged using a sequence-based method (the flow with the highest priority is taken as the unified flow. If the two flows have the same priority, the sizes are added together and the priority remains unchanged), and then return to S703;

[0101] S710. If there is no unmerged scheduled flow in L, that is, all flows have been merged, then output the unified flow list L uni .

[0102] 605. Unified stream selection and segmentation module, in this module, the unified stream in is selected and segmented according to the maximum number of streams that can be sent, and finally a list of actually sent streams is output. Figure 8 As shown, the logic of selecting and splitting the unified stream is shown:

[0103] The unified stream selection and segmentation module (605) specifically includes the following contents in the unified stream selection and segmentation:

[0104] S802. Receive the output L of the unified stream merging module (604) uni As input;

[0105] S803. Determine L act Whether the number of midstreams has reached the upper limit;

[0106] S804. If L act If the number of midstreams has not reached the upper limit, then L uni Are there any other items not included in L act The unified flow of

[0107] S805. If L uni If there are still flows not included in L uni Take out a flow with the highest rate or priority and add it to L act , return to S803;

[0108] S806. If L uni All have been taken out, then from L act Select a flow with the highest rate or priority;

[0109] S807. Determine whether the scheduling algorithm is rate-based;

[0110] S808. If the scheduling algorithm is rate-based, then the rate-based method is used to split the selected stream into two (the size and rate of the selected stream are equally divided into two new streams), and then return to S803;

[0111] S809. If the scheduling algorithm is not rate-based, it is determined to be sequence-based, and the selected stream is split into two using a sequence-based method (the size of the selected stream is equally divided into two new streams, and the priority of the two new streams remains the same as the selected stream), and then returns to S803;

[0112] S810. If L act The number of flows reaches the upper limit, the algorithm ends, and the list of flows actually emitted is output. act .

[0113] 606. Data transmission module, this module uses the unified stream queue L actually sent by module 605 act , send unified stream transmission data according to the information of this queue;

[0114] 607. Cluster computing data interaction module, this module uses the communication demand information transmitted by 602 to collect the data that the mapper needs to transmit or distribute the network data received by 608 to the reducer. The communication information of the sender (mapper) is usually the storage location of the data to be transmitted, and the information of the receiver (reducer) is usually the IP address and port number of the sender.

[0115] 608. Data transmission module, this module uses the unified stream queue actually sent finally given by module 605, and sends unified stream transmission data according to the information of this queue;

[0116] The interactive relationship in this embodiment is described as follows:

[0117] 609. The cluster computing framework transmits the communication requirements to the present invention;

[0118] 610. The communication requirements are passed to the co-flow scheduling algorithm, which uses this information to schedule the co-flows;

[0119] 611. The scheduling flow list is passed to the unified flow merging module;

[0120] 612. The merged unified stream is passed to the unified stream selection and segmentation module;

[0121] 613. The communication requirements are passed to the cluster computing data interaction module;

[0122] 614. Actual transmission of unified stream metadata;

[0123] 615. The cluster computing data interaction module interacts with the cluster computing framework to calculate data;

[0124] 616. Computational data interacts internally in the present invention;

[0125] 617. The data transmission module interacts with the data center network to transmit computing data.

[0126] In this embodiment, the unified communication agent reduces the total number of data flows in the data center network by proxying the communication process in the cluster computing framework, thereby reducing the queue length in the switch, thereby achieving the effect of reducing network latency, reducing packet loss rate, and improving network performance.

[0127] We verified the above effects through NS3 simulation experiments.

[0128] Network topology: Fig. 9 , Fig.12 The topology includes 1 switch connected to 16 hosts. The hosts and switches are connected with a bandwidth of 1Gbps. Fig.10 The topology contains 512 hosts. Fig.11 The topology is a Fattree topology with a base of 6.

[0129] Traffic between hosts: Based on one hour of co-flow sampling data from Facebook, it is generated by scaling according to bandwidth ratio. Fig.12 In the experiments presented, we scaled the traffic from two dimensions, time and size, to form four types of traffic: SI, SF, BI, and BF (small traffic arrives frequently, small traffic arrives infrequently, large traffic arrives frequently, and large traffic arrives infrequently).

[0130] Test indicators: average completion time and 99th percentile completion time of the co-flow, maximum buffer occupancy and packet loss rate of the switch.

[0131] Comparison algorithms: Hadoop's flow number limiting mechanism, the present invention (Courier), Aalo (a non-predictive co-flow scheduling algorithm) + Hadoop's flow number limiting mechanism, Aalo + the present invention (Courier), Sincronia (a predictive co-flow scheduling algorithm) + Hadoop's flow number limiting mechanism, Sincronia + the present invention (Courier), Django (a co-flow scheduling algorithm that limits the number of flows).

[0132] like Fig. 9 The average CCT, 99th percentile tail CCT, queue length, and packet loss rate in six scenarios were compared for different numbers of initiated flows per reducer (from 1 to 15, with a step size of 1). When the number of flows per reducer is relatively small (about 1-5), the difference in average CCT between Hadoop and Courier is not significant. This is because packet loss has not yet become a major factor affecting network performance. As the number of flows per reducer increases, Hadoop's average CCT continues to increase and packet loss becomes increasingly severe, while Courier's average CCT does not increase significantly. The performance of the 99th percentile tail CCT is similar to the average CCT, indicating that Courier does not cause a long tail effect. On average, Courier's average CCT is 22% lower than Hadoop's, and its 99th percentile tail CCT is 23% lower than Hadoop's. Fig. 9The third sub-graph reveals why Courier shortens CCT. Without Courier, Hadoop’s maximum buffer occupancy always reaches the switch’s buffer limit. Therefore, in Hadoop, the packet loss rate increases with the number of flows, leading to an increase in CCT. In contrast, Courier can effectively reduce buffer occupancy. Avoiding packet loss does not affect CCT.

[0133] like Fig. 9 The 4th to 6th sub-figures of show the evaluation of Aalo+Hadoop and Aalo+Courier. When the number of flows that can be started per reducer is small (about 1-2), the scheduling ability of Aalo is affected by the limitation of the number of flows. As the number of flows increases, the scheduling ability of Aalo gradually recovers, but the CCT of Aalo+Hadoop gradually increases due to the impact of packet loss on performance. In contrast, the CCT of Aalo+Courier gradually decreases as the flow increases in the absence of packet loss, reflecting the improvement of scheduling ability. The experimental results of average CCT and 99th percentile CCT are similar. The 99th percentile CCT is more unstable because it is dominated by a few large co-flows. Therefore, it mainly depends on how many times the large co-flows experience packet loss and retransmission, which is more accidental. The CCT of Aalo+Courier is always lower than that of Aalo+Hadoop (and Courier), proving that Courier can be well integrated with Aalo to take advantage of the scheduling ability of the scheduling algorithm.

[0134] like Fig. 9The performance of Sincronia+Hadoop and Sincronia+Courier shown in the last three sub-figures is similar to that of Aalo+Hadoop and Aalo+Courier shown above. Comparing these six cases, we can draw three conclusions. ① On average, the packet loss rates of Aalo+Hadoop and Hadoop are basically the same, and the loss rate of Sincronia+Hadoop is 167% of that of Hadoop. This is because the DSCP mechanism used by Sincronia actually reduces the available buffer space for each priority. Therefore, when the buffer limit has not been reached, the buffer occupancy rate of Sincronia+Courier is higher than that of Courier and Aalo+Courier. ② It can be found that the CCT of Sincronia+Hadoop (Sincronia+Courier) is lower than that of Aalo+Hadoop (Sincronia+Courier). This is because the predictability of Sincronia gives it stronger scheduling capabilities than Aalo. In addition, we can clearly see that the CCT of Sincronia+Courier is lower than that of the other five cases, and the CCT continues to decrease as the traffic increases. It can be seen that combining the Courier algorithm with the scheduling algorithm can avoid the negative impact of packet loss and give full play to the scheduling capability. ③ In all experiments, Courier effectively controlled the queue length, avoided packet loss, reduced CCT, and avoided the long tail in CCT. This shows that Courier can control the number of concurrent flows in the network, thereby providing better network performance.

[0135] like Fig.10 The simulation results under the Fat tree topology with k=6 are given. We compare the performance of Hadoop and Courier under different numbers of flows between each pair of servers. In this experiment, we deploy two multipath routing methods, ECMP and XPath, respectively. The horizontal axis represents the number of flows of Courier, so the CCT and loss rate of Hadoop are represented by the horizontal line. It can be observed that when the number of flows increases to 4 or more, the CCT of Courier is lower than that of Hadoop regardless of which multipath method is used. When the number of flows between each pair of servers is less than 6, the packet loss rate of Courier is lower than that of Hadoop. When the number of flows is greater than 6, the packet loss rate of Courier is higher than that of Hadoop, because Courier artificially utilizes more paths (initiates more flows than Hadoop). However, even with a higher packet loss rate, Courier's CCT is lower than that of Hadoop because Courier utilizes more paths.

[0136] like Fig.11 Figure 1 shows our evaluation of Courier in a large topology of 512 servers. As in the previous experiments, the horizontal axis represents the number of flows that a server can start in Courier. Therefore, Hadoop's CCT and loss rate are shown as a horizontal line, where the number of flows that a reducer can start is set to 5 (the default in Hadoop). When the number of flows that each server can start is too small, Courier's average CCT increases due to head-of-line blocking. When too many flows can be started per server, Courier's average CCT increases due to increased packet loss. When the number of flows per server is between 5 and 15, Courier effectively balances the two factors of head-of-line blocking and packet loss, reducing the average CCT. In comparison, Hadoop's average CCT and packet loss rate are not as good as Courier because it cannot effectively limit the number of flows.

[0137] like Fig.12 , observing the average CCT of Django in the figure, we can find that Django performs the worst under SF workload (almost the same as Hadoop). This is because Django's flow number limit mechanism blocks small co-flows. The smaller the size of the co-flow, the more significant the impact of blocking, and the more frequently the co-flows arrive, the greater the chance of blocking. This also explains why Django performs best under BI workload. Averaging the performance of all workloads, Django's average CCT is lower than Aalo+Hadoop, but basically the same as Aalo+Courier. However, comparing the performance of Django and Sincronia in the average of all workloads, we can observe that Sincronia+Hadoop's CCT is 8% higher than Django. But with the help of Courier, Sincronia+Courier performs 34% better than Django in the average of all workloads. 99th percentile tail CCT ( Fig.12 The second subgraph of ) shows a similar trend as the average CCT, where Django performs worst in the SF workload, best in the BI workload, and worse than Sincronia+Courier on average for all workloads.

[0138] These results show that 1) Django successfully avoids packet loss by limiting the number of flows, thereby achieving a lower CCT than ordinary scheduling algorithms. 2) However, Django's scheduling-based flow number restriction mechanism itself limits its scheduling capabilities. By combining with Courier, mainstream algorithms can achieve better results than Django.

[0139] The difference between this embodiment and the prior art is that almost all existing co-flow scheduling algorithms do not take into account the negative impact that the number of concurrent flows may have on the data center network. Hadoop uses a simple flow number limitation mechanism to try to solve this problem, but this mechanism will cause head-of-line blocking and affect the scheduling effect of the co-flow scheduling algorithm. Django, as a co-flow scheduling algorithm that takes the number of concurrent flows into consideration, has a core defect in the head-of-line blocking effect on co-flows caused by limiting the number of flows. The present invention reduces the number of flows in the network and improves network performance by proxying flows in cluster computing and using unified flows instead of scattered flows.

[0140] The present invention discloses a network flow proxy method in cluster computing. By proxying the communication demands in cluster computing, the communication demands are integrated according to source-destination address pairs, the number of flows in cluster computing is reduced, the network performance is improved, and the original scheduling attributes are maintained when the flows are merged. The method can be well coupled with the existing co-flow scheduling algorithm to reduce the co-flow completion time. In consideration of the complexity of the actual topology of a data center, a max-min fairness mechanism is used to adapt to the topology of the data center to the maximum extent while maintaining the scheduling order when the number of flows allowed to be initiated is limited.

[0141] By using the present invention, the number of concurrent flows can be effectively reduced, thereby reducing packet loss and CCT in various scenarios and topologies.

[0142] The present invention integrates well with mainstream scheduling algorithms and brings an average CCT speedup of 16% to 29% in all workloads.

[0143] The performance of the present invention combined with the mainstream scheduling algorithm is about 30% higher than that of the most advanced scheduling-based flow quantity control method (Django).

[0144] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A unified communication agent supporting concurrent flow scheduling in cluster computing, It is characterized in that By deploying a unified communication agent on the machines participating in cluster computing, the number of flows between two pairs of servers is reduced to one, including: The product form of the unified communication agent is unified communication agent software (401), which supports the function of receiving the communication demand of the cluster computing framework and transmitting data on its behalf. The unified communication agent software (401) includes a communication demand processing module (402) and a data transmission module (403); the communication demand processing module (402) will pass the communication demand to the data transmission module (403), and the data transmission module (403) will uniformly initiate a data flow to transmit data according to the communication demand; S502, receiving communication requirements of a cluster computing framework; S503: For all communication requirements, run the co-flow scheduling algorithm to calculate their appropriate transmission rates or priorities; S504, calculating the transmission rate or priority of the unified flow to each other server according to the result of the co-flow scheduling algorithm; S505, transmitting data through a unified stream; Wherein step S502, step S503 and step S504 are completed by the control module (601), and step S505 is completed by the data module (606), the control module (601) is equivalent to the demand processing module (402) in the unified communication agent software (401), and the data module (606) is equivalent to the data transmission module (403) in the unified communication agent software (401); The step S503 specifically includes: running the existing co-flow scheduling algorithm through the co-flow scheduling algorithm module (603), taking the communication requirements of the cluster computing framework received in step S502 as input, and outputting a scheduled data flow list L, wherein each flow is defined as a four-tuple <address pair, size, rate, priority>; the rate field is only valid when the deployed co-flow scheduling algorithm is a rate-based scheduling algorithm, and the same applies to the priority field; The step S504 specifically includes: using the unified stream merging module (604) to take the data stream list L outputted in step S503 as input, merging the scheduled streams in the data stream list L into a unified stream, and outputting the unified stream list ; Then, the unified stream list is selected and segmented by the unified stream selection and segmentation module (605). The unified stream in is selected and split according to the maximum number of streams that can be sent, and finally the list of actually sent streams is output. ; The step S505 specifically includes: transmitting the data of the cluster computing task according to the information calculated by the control module (601) through the data module (606), which is specifically completed by the cluster computing data interaction module (607) and the data transmission submodule (608), wherein the cluster computing data interaction module (607) uses the communication demand information transmitted by the communication demand receiving module (602) to collect the data that the mapper sender needs to transmit or distributes the network data received by the data transmission submodule (608) to the reducer receiver, the communication information of the mapper sender is the storage location of the data to be transmitted, and the information of the reducer receiver is the IP address and port number of the sender; the data transmission submodule (608) uses the unified flow queue actually sent finally given by the unified flow selection and segmentation module (605) to send the unified flow transmission data according to the information of this queue.

2. A unified communication agent supporting concurrent flow scheduling in cluster computing according to claim 1, It is characterized in that The step S502 specifically includes: receiving the communication demand of the cluster computing framework through the communication demand receiving module (602).

3. A unified communication agent supporting concurrent flow scheduling in cluster computing according to claim 1, It is characterized in that The unified stream merging module (604) merges the scheduled streams into a unified stream, specifically including the following contents: S702. Receive the data stream list L output by the co-stream scheduling algorithm module (603) as input; S703. Determine whether there are any unmerged scheduling flows in L; S704. If there are still unmerged scheduled flows in L, one of them is taken out as the flow to be merged; S705. Determine the unified flow list whether there is a unified flow corresponding to the flow retrieved in S704; S706. If there is no corresponding unified flow, create an empty unified flow and put it into the unified flow list ; S707. Determine whether the scheduling algorithm is rate-based; S708. If the scheduling algorithm is rate-based, then the rate-based method is used to merge the extracted flow and the unified flow, and return to S703; S709. If the scheduling algorithm is not rate-based, it is determined to be sequence-based, and the two streams are merged using a sequence-based method, and then the algorithm returns to S703; S710. If there is no unmerged scheduled flow in L, that is, all flows have been merged, then output a unified flow list .

4. A unified communication agent supporting concurrent flow scheduling in cluster computing according to claim 1, It is characterized in that The unified stream selection and segmentation module (605) specifically includes the following contents in unified stream selection and segmentation: S802. Receive the output of the unified stream merging module (604) As input; S803. Judgment Whether the number of midstreams has reached the upper limit; S804. If If the number of midstreams has not reached the upper limit, then Are there any other The unified flow of S805. If If there are still flows not included in Take out a flow with the highest rate or priority and add it , return to S803; S806. If All have been taken out, then from Select a flow with the highest rate or priority; S807. Determine whether the scheduling algorithm is rate-based; S808. If the scheduling algorithm is rate-based, then the rate-based method is used to split the selected flow into two, and return to S803; S809. If the scheduling algorithm is not rate-based, it is determined to be sequence-based, and the selected flow is split into two using a sequence-based method, and returns to S803; S810. If The number of flows reaches the upper limit, the algorithm ends, and the list of flows actually emitted is output .