A Coflow scheduling system based on TD3 algorithm

Abstract

The present invention discloses a Coflow scheduling system based on the TD3 algorithm. The system includes a memory, a processor, and a computer program module stored in the memory and executable on the processor. The system comprises: a preprocessing module for extracting and preprocessing Coflow information in a distributed machine learning communication environment; a Markov model construction module for establishing a Markov model based on Coflow deep reinforcement learning, defining actions, environmental states, and reward functions; a model training module for repeatedly iteratively training the Coflow environment using the deep reinforcement learning TD3 algorithm to obtain a final model and policy function; and a communication module for communicating Coflows in the distributed machine learning using the obtained model and policy function. The present invention can adaptively adjust the thresholds of multi-level feedback queues to better adapt to different distributed application scenarios, thereby expanding the scope of application and improving scheduling efficiency.

Description

Technical Field

[0001] The present invention relates to a Coflow queue scheduling technology, in particular to a Coflow scheduling system based on a TD3 algorithm. Background Art

[0002] In recent years, due to the exponential growth in data volume and the complexity of machine learning in data center networks, a single computer can no longer meet the required computing performance. Therefore, distributed machine learning is running more and more widely. However, there are many problems with the information exchange between different machines, and the use of a single Flow has defects such as complex description.

[0003] To address the shortcomings of a single flow, researchers proposed the concept of Coflow. These applications typically generate a set of parallel flows from mapper to reducer, called a Coflow. Each individual flow in the set is called a subflow of the Coflow. Flows in a Coflow are semantically related to each other and have a common goal of collaborative completion. When all flows belonging to a Coflow are completed, the Coflow is considered complete.

[0004] Coflow scheduling optimization methods primarily include Varys, Aalo, and CODA. Research focuses on centralized vs. distributed approaches, prior knowledge vs. no prior knowledge, and efficiency vs. fairness priorities. Varys, first proposed in 2014, relies on prior knowledge, while subsequent approaches like Aalo and CODA are knowledge-free.

[0005] The main idea of ​​Aalo is to use Multi-Level Feedback Queuing (MLFQ) to prioritize coflows and implement scheduling. MLFQ uses a fixed threshold to prioritize coflows based on size and assigns them to different priority queues. Different queues use CLAS scheduling, which is sensitive to coflow size, and within the same queue, a first-come, first-served (FIFO) approach.

[0006] Because MLFQ typically uses fixed thresholds, it's not suitable for all scenarios. This is especially true for distributed machine learning scenarios, where data types vary and system flexibility is critical. Therefore, it's necessary to convert fixed thresholds into variable thresholds to improve MLFQ's robustness. Manually adjusting thresholds dynamically, while effective, is difficult due to the long optimization cycle and heavy workload. Summary of the Invention

[0007] The purpose of the present invention is to provide a Coflow scheduling system based on the TD3 algorithm, which adaptively optimizes the threshold of the Coflow multi-level feedback queue and can automatically adjust the threshold according to different usage scenarios to reduce the average Coflow completion time and thus improve the communication efficiency between distributed machine learning.

[0008] The technical solution for achieving the purpose of the present invention is: a Coflow scheduling system based on the TD3 algorithm, comprising a memory, a processor, and a computer program module stored in the memory and executable on the processor, wherein the computer program module comprises a preprocessing module, a Markov model construction module, a model training module, and a communication module arranged in sequence, wherein:

[0009] The preprocessing module is used to extract Coflow information in a distributed machine learning communication environment and preprocess the information;

[0010] The Markov model building module is used to build a Markov model based on Coflow deep reinforcement learning, defining actions, environment states, and reward functions;

[0011] The model training module is used to repeatedly iteratively train the Coflow environment using the deep reinforcement learning TD3 algorithm to obtain the final model and policy function;

[0012] The communication module is used to communicate Coflow in distributed machine learning using the model and strategy function obtained by the model training module.

[0013] Compared with the prior art, the present invention has the following significant advantages:

[0014] (1) Improving the fixed-threshold multi-level feedback queue to an adaptive-threshold multi-level feedback queue can be applied to a wider range of scenarios, improve the robustness of the system, and effectively reduce communication transmission time and transmission overhead;

[0015] (2) A Coflow-based reward function is proposed, which can effectively provide feedback on the effects of the actions adopted and improve the efficiency of system training;

[0016] (3) The environment of Coflow is modeled and the deep reinforcement learning TD3 algorithm is used. The branch reduction method is used to explore a wider range of actions per unit time, and the use of a parallel critic network can avoid the problem of over-estimation; the use of delayed updates can stabilize the Q value in Coflow and reduce erroneous updates. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of Coflow multi-level feedback queue.

[0018] Figure 2 This is the network architecture diagram of the TD3 algorithm.

[0019] Figure 3 This is the Coflow scheduling flow chart based on the TD3 algorithm.

[0020] Figure 4 This is the Actor neural network structure diagram.

[0021] Figure 5 This is the structural diagram of the Critic neural network.

[0022] Figure 6 This is a comparison chart of the average completion time of Coflow. DETAILED DESCRIPTION

[0023] It is easy to understand that, based on the technical solution of the present invention, those skilled in the art can imagine various embodiments of the present invention without changing the essential spirit of the present invention. Therefore, the following specific embodiments and drawings are merely illustrative of the technical solution of the present invention and should not be regarded as the entire invention or as limiting or defining the technical solution of the present invention.

[0024] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0025] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0026] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0027] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0028] The present invention provides a Coflow scheduling system based on the TD3 algorithm, comprising a memory, a processor, and a computer program module stored in the memory and executable on the processor, wherein the computer program module comprises a preprocessing module, a Markov model construction module, a model training module, and a communication module arranged in sequence, wherein:

[0029] The preprocessing module is used to extract Coflow information in a distributed machine learning communication environment and preprocess the information;

[0030] The Markov model building module is used to build a Markov model based on Coflow deep reinforcement learning, defining actions, environment states, and reward functions;

[0031] The model training module is used to repeatedly iteratively train the Coflow environment using the deep reinforcement learning TD3 algorithm to obtain the final model and policy function;

[0032] The communication module is used to communicate Coflow in distributed machine learning using the model and strategy function obtained by the model training module.

[0033] As a specific example, the preprocessing module preprocesses the information, including: extracting the size, number of flows included, arrival time, and duration of the Coflow information obtained.

[0034] As a specific example, the Markov model building module includes an action definition unit, an environment state definition unit, and a reward function definition unit, where:

[0035] The action definition unit is used to define the action in the Coflow deep reinforcement learning, and put each threshold of the multi-level feedback queue into the same set {TH1, TH2, ..., TH n}, to directly adjust the multi-level feedback queue threshold as an action;

[0036] The environment state definition unit is used to define the environment state in the Coflow-based deep reinforcement learning, and represents the size of the Coflow, the number of flows included, the arrival time, and the duration as the state information of the environment in the form of a multidimensional array;

[0037] The reward function definition unit is used to define a reward function based on Coflow deep reinforcement learning to score the actions implemented in the current environment.

[0038] As a specific example, the model training module is used to perform repeated iterative training on the Coflow environment using the deep reinforcement learning TD3 algorithm, specifically including:

[0039] A policy function initialization unit, used to first set the policy function parameters to default values;

[0040] An environment state initialization unit, used to select information at an initial moment from the environment state as current environment state information;

[0041] The environment state update unit is used to use the TD3 algorithm to process the current environment state information using a deep neural network, generate corresponding actions for the current environment state according to the policy function, and generate a new environment state;

[0042] The action evaluation unit is used to calculate the corresponding reward function value according to the change of the environmental state, and evaluate and score the implemented action to evaluate the effect of the action;

[0043] A policy function adjustment unit is used to adjust the parameters of the policy function according to the evaluation results of the action and the changes in the environment state;

[0044] The iteration unit repeatedly executes the environment state initialization unit, the environment state update unit, the action evaluation unit, and the strategy function adjustment unit until the average completion time of Coflow reaches the set value.

[0045] As a specific example, the communication module is used to communicate Coflow in distributed machine learning using the model and strategy function obtained by the model training module, specifically including:

[0046] In a distributed machine learning environment, the model and strategy function obtained by the model training module are used to schedule the communication of the transmitted Coflow information. By sampling the data in the network environment, the priority threshold of the multi-level feedback queue is adjusted in real time.

[0047] As a specific example, the action definition unit is used to define the action based on Coflow deep reinforcement learning, placing each threshold of the multi-level feedback queue in the same set {TH1, TH2, ..., TH n}, to directly adjust the multi-level feedback queue threshold as an action, as follows:

[0048] In the Coflow multi-level feedback queue, from top to bottom, the lowest priority Q K To the highest priority Q1, use the method of dynamically changing the threshold to put the threshold of each priority queue into a set, that is, {TH1, TH2, ..., TH n}, where the priority level Q1 corresponds to a range from 0 to TH1, the priority level Q2 corresponds to a range from TH1 to TH2, and so on, until the lowest priority level Q K , the corresponding value range is from TH n To ∞; by threshold {TH1,TH2,…,TH n} to adjust the entire multi-level feedback queue;

[0049] In a multi-level feedback queue, the newly arrived Coflow will first enter the highest priority queue. Then, as the size of the sent data increases, it will be downgraded according to the threshold of each priority, that is, each queue has a maximum threshold and a minimum threshold. That is, when the size of the sent data is greater than the first threshold TH1, the Coflow will be downgraded to the second priority queue. When the size of the sent data is greater than the second threshold TH2, the Coflow will be further downgraded to the third priority queue, and so on, until all the Coflow data is sent.

[0050] Each priority has a different weight in network transmission. The higher the weight of the priority, the higher the network bandwidth allocated; the lower the weight of the priority, the lower the network bandwidth allocated. The threshold of the multi-level feedback queue can dynamically adjust the queue size of different priorities, thereby controlling data transmission.

[0051] As a specific example, the environment state definition unit is used to define the environment state in Coflow-based deep reinforcement learning, and represents the size of Coflow, the number of flows included, the arrival time, and the duration as the state information of the environment in the form of a multidimensional array, as follows:

[0052] {Size1,Number1,Arrival_Time1,Duration1,Size2,…}

[0053] Among them, each four elements represent the information of a Coflow, and then the sorting operation is performed based on the arrival time, and the arrival time is sorted from small to large;

[0054] During operation, the order of entering the multi-level feedback queue is determined according to the sequence, and the queue where the Coflow is located is adjusted at the next moment; if the transmitted data is greater than the maximum threshold of the queue, the priority is lowered.

[0055] As a specific example, the reward function definition unit is used to define a reward function based on Coflow deep reinforcement learning to score the actions implemented in the current environment, as follows:

[0056] Set the reward function formula:

[0057]

[0058] in, is the continuous transmission time of the nth Coflow at time t and t+1; the difference between the continuous transmission time from time t+1 to time t is made, and the average of all N Coflows is calculated to reflect the change in the average completion time of the Coflow. Finally, the average value is normalized by the time difference to eliminate the influence of the time interval on the result; when a Coflow has completed the transmission, the latter term The result is 0.

[0059] As a specific example, the policy function is based on the actor-critic model, namely the Actor-Critic model, and the deep reinforcement learning algorithm adopted is the double-delay deep deterministic policy gradient algorithm, namely the TD3 algorithm, which includes a total of six neural networks, of which two neural networks belong to the Actor neural network, namely the Actor network and the Actor target network, and four neural networks belong to the Critic neural network, namely Critic network 1, Critic network 2, Critic target network 1, and Critic target network 2;

[0060] Among them, the Actor neural network is used to generate the corresponding action according to the state, and the Critic neural network is used to evaluate the action taken.

[0061] As a specific example, each neural network has four layers of neurons, namely the input layer, the hidden layer, and the output layer;

[0062] (1) Based on the starting state State, an Action is generated according to the neural network: {TH1, TH2, ..., TH n};

[0063] First, the defined starting state is used as the input of the Actor neural network. The Actor neural network will generate a one-bit vector, which is the action generated by the current state, that is, the threshold of a multi-level feedback queue. According to the generated action, the corresponding threshold of the multi-level feedback queue in the environment network is set, and the threshold of the multi-level feedback queue is updated again in the next stage.

[0064] The Actor network updates φ by increasing the policy gradient value, as follows:

[0065]

[0066] In the above formula, J(φ) is the loss function of Actor, and the right side of the equation represents the average of the sample gradient, where φ represents the parameters of the Actor network, θ1 represents the parameters of the Critic network, s represents the state, a represents the action, and Qθ1 (s,a) represents the output of the Critic network, π φ (s) is the output of the Actor network;

[0067] The critic network is updated using the following formula:

[0068]

[0069] In the above formula, θ i represents the parameters of the i-th Critic network, y represents the target Q value, Represents the output result of the i-th Critic network;

[0070] For the parameters φ of the Actor network and θ1 of the Critic network, the parameters of the Actor target network and the Critic target network are soft-updated according to the following two formulas:

[0071] θ′ i ←τθ i +(1τ)θ′ i

[0072] φ′←τφ+(1-τ)φ′

[0073] Where τ represents the update step size;

[0074] Use the experience replay pool to record decision information and set the maximum capacity to 10,000;

[0075] (2) Use the Critic neural network to score the current {State, Action} using the reward function;

[0076] The function of the Critic neural network is to judge the effect of the current action under the current state. In the TD3 algorithm, a delayed update strategy is used. That is, the Actor network will only be updated once after the Critic neural network has been updated multiple times, making the training of the Actor neural network more stable.

[0077] (3) Repeat steps (1) to (2) until the fixed number of cycles is reached or the average completion time of the Coflow is less than the specified time, and then end the cycle.

[0078] The present invention is described clearly and completely below with reference to the accompanying drawings and specific embodiments of the present invention.

[0079] Example 1

[0080] In this embodiment, the distributed machine learning uses the Spark framework, which includes 150 computing nodes.

[0081] In order to solve the problem of low efficiency and poor compatibility when using multi-level feedback queues to schedule Coflow, this embodiment provides a Coflow scheduling method based on the TD3 algorithm, combined with Figures 1 to 3 , its main steps are as follows:

[0082] Step 1: Extract Coflow information in the distributed machine learning communication environment and preprocess the information;

[0083] Step 2: Build a Markov model based on Coflow deep reinforcement learning, defining actions, environment states, and reward functions;

[0084] Step 3: Use the deep reinforcement learning TD3 algorithm to repeatedly iteratively train the Coflow environment to obtain the ideal model results and policy function;

[0085] Step 4: Use the obtained model and policy function to communicate Coflow in distributed machine learning;

[0086] The present invention will be further described below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0087] The establishment of the Markov model described in step 2 specifically includes:

[0088] (1) Define the actions in Coflow-based deep reinforcement learning, and put each threshold of the multi-level feedback queue into the same set {TH1, TH2, ..., TH n}, thereby directly adjusting the multi-level feedback queue threshold as an action;

[0089] exist Figure 1 In the middle, from top to bottom, the lowest priority is Q K To the highest priority Q1, different from the previous fixed threshold multi-level feedback queue, the present invention uses a method of dynamically changing the threshold. The threshold of each priority queue is placed in a set, namely {TH1, TH2, ..., TH n}, where the priority level Q1 corresponds to a range from 0 to TH1, the priority level Q2 corresponds to a range from TH1 to TH2, and so on, until the lowest priority level Q K , the corresponding value range is from TH n To ∞. By adjusting the threshold {TH1,TH2,…,TH n} can be adjusted to adjust the entire multi-level feedback queue.

[0090] In this embodiment, n is set to 9, and the priority queue thresholds are {TH1, TH2, …, TH9}, that is, there are 10 priority queues, namely 0~TH1, TH1~TH2, TH2~TH3, TH3~TH4, TH4~TH5, TH5~TH6, TH6~TH7, TH7~TH8, TH8~TH9, and TH9~∞.

[0091] In a multi-level feedback queue, a newly arrived Coflow will first enter the highest priority queue. Then, as the size of the data it sends is determined by the threshold of each priority, that is, each queue has a maximum threshold and a minimum threshold, it will be downgraded. That is, when the size of the data it sends is greater than the first threshold TH1, the Coflow will be downgraded to the second priority queue. If the size of the data it sends is greater than the second threshold TH2, it will be further downgraded to the third priority queue, and so on, until all the data of the Coflow is sent.

[0092] Each priority has a different weight in network transmission. The higher the weight, the higher the network bandwidth allocated to the priority; the lower the weight, the lower the network bandwidth allocated to the priority. The threshold of the multi-level feedback queue can dynamically adjust the queue size of different priorities, thereby controlling data transmission.

[0093] (2) Define the environment state in Coflow-based deep reinforcement learning, and express the size of Coflow, including the number of flows, arrival time, duration, etc., as the state information of the environment in the form of a multidimensional array, as shown below:

[0094] {Size1,Number1,Arrival_Time1,Duration1,Size2,…}

[0095] Each of the four elements represents a Coflow, which is then sorted by arrival time, from smallest to largest. At runtime, this order determines the order in which Coflows enter the multi-level feedback queue, adjusting the queue at each subsequent moment. If the size of the transmitted data exceeds the maximum threshold of its queue, its priority is lowered.

[0096] In this embodiment, the environment state of Coflow is set to 10 groups of Coflow information, each Coflow includes

[0097] 4 elements, so the environment state includes a total of 40 elements, namely:

[0098] {Size1,Number1,…,Size 10 ,Number10 ,Arrival_Time 10 ,Duration 10}

[0099] (3) Define a reward function based on Coflow deep reinforcement learning to score the actions taken in the current environment;

[0100] Set the reward function formula:

[0101]

[0102] in, The duration of continuous transmission of the nth Coflow at time t is subtracted from the duration of continuous transmission from time t+1 to time t, and the average of all N Coflows is calculated to reflect the change in the average completion time of the Coflow. Finally, the time difference is normalized to eliminate the influence of time interval on the result. When a Coflow has completed transmission, the latter term The result is 0.

[0103] In the Coflow that has been transmitted in this embodiment, its duration t+1 =duration t , so the value of Reward is 0.

[0104] Furthermore, in step 3, the deep reinforcement learning TD3 algorithm is used to repeatedly iteratively train the Coflow environment. Using the established Markov model, the existing Coflow-related data is selected for training, repeated iterations are performed, and finally the trained model, policy function, etc. are saved. The details are as follows:

[0105] In this embodiment, the policy function used is based on the Actor-Critic model, and the deep reinforcement learning algorithm adopted is the TD3 algorithm (Twin Delayed Deep Deterministic Policy Gradient, Chinese full name: Double Delayed Deep Deterministic Policy Gradient), which includes a total of six neural networks, such as Figure 2 As shown in the figure, there are two networks belonging to the Actor neural network, namely the Actor network and the Actor target network, and the other four networks belong to the Critic neural network, namely Critic network 1, Critic network 2, Critic target network 1, and Critic target network 2.

[0106] Among them, the Actor neural network is used to generate corresponding actions according to the state State, such as Figure 4 As shown; Critic neural network is used to evaluate the action taken, such as Figure 5 As shown in Figure 2. All neural networks have four layers of neurons: input layer, hidden layer, and output layer.

[0107] (1) Based on the starting state State, an Action is generated according to the neural network: {TH1, TH2, ..., TH n};

[0108] Specifically, first, the defined starting state is used as the input of the Actor neural network.

[0109] A one-bit vector is generated, representing the action generated by the current state, i.e., the threshold of a multi-level feedback queue. Based on the generated action, the corresponding threshold of the multi-level feedback queue in the environment network is set, and the threshold of the multi-level feedback queue is updated again in the next phase. In this embodiment, the Actor network has 40 input layer neurons, 64 hidden layer neurons, and 9 output layer neurons.

[0110] Among them, the Actor network increases the policy gradient value, and its formula is as follows:

[0111]

[0112] In the above formula, φ represents the parameters of the Actor network, and θ1 represents the parameters of the Critic network.

[0113] The critic network is updated using the following formula:

[0114]

[0115] Since there will be two identical Critic networks updating the same target, but the initial values ​​of the network parameters are different, the final calculated results will be different, so a lower value can be selected to estimate the Q value to avoid the Q value being overestimated.

[0116] For the parameters φ of the Actor network and θ1 of the Critic network, the parameters of the Actor target network and the Critic target network are soft-updated according to the following two formulas:

[0117] θ′ i ←τθ i +(1-τ)θ′ i

[0118] φ′←τφ+(1-τ)φ′

[0119] Here, τ represents the update step size, which is 0.01 in this embodiment. An experience replay pool is used to record decision information, and its maximum capacity is set to 10,000 in this embodiment.

[0120] (2) Use the Critic neural network to score the current {State, Action} using the reward function;

[0121] Its neural network structure is as follows Figure 5 As shown, specifically, its function is mainly to judge the effect of using the current action Action under the current state State. In this embodiment, the number of neurons in the hidden layer of the Critic neural network used is 64, and the number of neurons in the output layer is 1.

[0122] In the TD3 algorithm, a delayed update strategy is used, that is, the Actor network will only be updated once after the Critic neural network has been updated multiple times, making the training of the Actor neural network more stable.

[0123] (3) Repeat steps (1) and (2) until a fixed number of cycles is reached, or the average completion time of Coflow is reached.

[0124] When the time is less than the specified time, the cycle ends.

[0125] Furthermore, the obtained model and policy function are used in step 4 to test the data obtained in the network environment, and the evaluation criterion is the average completion time of Coflow.

[0126] In order to compare the effect of the present invention, this embodiment uses different Coflow scheduling algorithms to compare the average Coflow completion time on the same data set. There are three different data sets, such as Figure 6 As shown in the three different data sets, using the average Coflow completion time of our method as a benchmark, the average Coflow completion time of SCF and NCF is higher than the baseline, while the average Coflow completion time of SEBF is lower than the baseline. Therefore, we can conclude that our scheduling method outperforms SCF and NCF, but slightly underperforms the globally known optimal SEBF scheduling algorithm.

[0127] Example 2

[0128] This embodiment provides a Coflow scheduling system based on the TD3 algorithm, including a memory, a processor, and a computer program module stored in the memory and executable on the processor. The computer program module includes a preprocessing module, a Markov model construction module, a model training module, and a communication module, which are arranged in sequence.

[0129] The preprocessing module is used to extract Coflow information in a distributed machine learning communication environment and preprocess the information;

[0130] The Markov model building module is used to build a Markov model based on Coflow deep reinforcement learning, defining actions, environment states, and reward functions;

[0131] The model training module is used to repeatedly iteratively train the Coflow environment using the deep reinforcement learning TD3 algorithm to obtain the final model and policy function;

[0132] The communication module is used to communicate Coflow in distributed machine learning using the model and strategy function obtained by the model training module.

[0133] The processor described in this embodiment implements the Coflow scheduling method based on the TD3 algorithm in Example 1 when executing the computer program module.

[0134] Example 3

[0135] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps in the Coflow scheduling method based on the TD3 algorithm in Example 1 are implemented.

[0136] In summary, the present invention combines the TD3 algorithm and the multi-level feedback queue to adaptively adjust the threshold of the multi-level feedback queue. Compared with the traditional multi-level feedback queue with a fixed threshold, it can better adapt to different application scenarios and effectively improve the scope of application and scheduling efficiency.

[0137] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

[0138] It should be understood that in order to simplify the present invention and help those skilled in the art understand the various aspects of the present invention, in the above description of the exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or described with reference to a single figure. However, the present invention should not be interpreted as if all the features included in the exemplary embodiments are essential technical features of the claims of this patent.

Claims

1. A Coflow scheduling system based on the TD3 algorithm, characterized in that: The system comprises a memory, a processor, and a computer program module stored in the memory and executable on the processor, wherein the computer program module comprises a pre-processing module, a Markov model construction module, a model training module, and a communication module arranged in sequence, wherein: The preprocessing module is used to extract Coflow information in a distributed machine learning communication environment and preprocess the information; The Markov model building module is used to build a Markov model based on Coflow deep reinforcement learning, defining actions, environment states, and reward functions; The model training module is used to repeatedly iteratively train the Coflow environment using the deep reinforcement learning TD3 algorithm to obtain the final model and policy function; The communication module is used to communicate Coflow in distributed machine learning using the model and strategy function obtained by the model training module; The TD3 algorithm includes six neural networks, each of which has four layers of neurons, namely the input layer, the hidden layer and the output layer. (1) Based on the starting state State, generate an Action according to the neural network: {TH 1, TH2, … , TH n }; First, the defined starting state is used as the input of the Actor neural network. The Actor neural network will generate a one-bit vector, which is the action generated by the current state, that is, the threshold of a multi-level feedback queue. According to the generated action, the corresponding threshold of the multi-level feedback queue in the environment network is set, and the threshold of the multi-level feedback queue is updated again in the next stage. The Actor network is updated by increasing the policy gradient value , the formula is as follows: ； In the above formula, is the loss function of the Actor, and the right side of the equation represents the average of the sample gradients, where Represents the parameters of the Actor network, Represents the parameters of the Critic network, Indicates status, Indicates action, Represents the output of the Critic network, Is the output of the Actor network; The critic network is updated using the following formula: ； In the above formula, Indicates the The parameters of the critic network, Indicates the target value, Indicates the The output of a critic network; Parameters for the Actor Network and the parameters of the Critic network , soft update the parameters of the Actor target network and the Critic target network according to the following two formulas: ； ； in It represents the update step size; Use the experience replay pool to record decision information and set the maximum capacity to 10,000; (2) Use the Critic neural network to score the current {State, Action} using the reward function; The function of the Critic neural network is to judge the effect of the current action under the current state. In the TD3 algorithm, a delayed update strategy is used. That is, the Actor network will only be updated once after the Critic neural network has been updated multiple times, making the training of the Actor neural network more stable. (3) Repeat steps (1) to (2). When the fixed number of cycles is reached or the average completion time of Coflow is less than the specified time, the cycle process ends.

2. The Coflow scheduling system based on the TD3 algorithm according to claim 1 is characterized in that: The preprocessing module preprocesses the information, including: extracting the size, number of flows included, arrival time, and duration of the Coflow information obtained.

3. The Coflow scheduling system based on the TD3 algorithm according to claim 1 is characterized in that: The Markov model building module includes an action definition unit, an environment state definition unit, and a reward function definition unit, where: The action definition unit is used to define the action in the Coflow deep reinforcement learning, placing each threshold of the multi-level feedback queue in the same set. 1, TH2,…,TH n }, to directly adjust the multi-level feedback queue threshold as an action; The environment state definition unit is used to define the environment state in the Coflow-based deep reinforcement learning, and represents the size of the Coflow, the number of flows included, the arrival time, and the duration as the state information of the environment in the form of a multidimensional array; The reward function definition unit is used to define a reward function based on Coflow deep reinforcement learning to score the actions implemented in the current environment.

4. The Coflow scheduling system based on the TD3 algorithm according to claim 1 is characterized in that: The model training module is used to perform repeated iterative training on the Coflow environment using the deep reinforcement learning TD3 algorithm, specifically including: A policy function initialization unit, used to first set the policy function parameters to default values; An environment state initialization unit, used to select information at an initial moment from the environment state as current environment state information; The environment state update unit is used to use the TD3 algorithm to process the current environment state information using a deep neural network, generate corresponding actions for the current environment state according to the policy function, and generate a new environment state; The action evaluation unit is used to calculate the corresponding reward function value according to the change of the environmental state, and evaluate and score the implemented action to evaluate the effect of the action; A policy function adjustment unit is used to adjust the parameters of the policy function according to the evaluation results of the action and the changes in the environment state; The iteration unit repeatedly executes the environment state initialization unit, the environment state update unit, the action evaluation unit, and the strategy function adjustment unit until the average completion time of Coflow reaches the set value.

5. The Coflow scheduling system based on the TD3 algorithm according to claim 1 is characterized in that: The communication module is used to communicate Coflow in distributed machine learning using the model and strategy function obtained by the model training module, specifically including: In a distributed machine learning environment, the model and strategy function obtained by the model training module are used to schedule the communication of the transmitted Coflow information. By sampling the data in the network environment, the priority threshold of the multi-level feedback queue is adjusted in real time.

6. The Coflow scheduling system based on the TD3 algorithm according to claim 3 is characterized in that: The action definition unit is used to define the action in the Coflow deep reinforcement learning, placing each threshold of the multi-level feedback queue in the same set. 1, TH2,…,TH n }, to directly adjust the multi-level feedback queue threshold as an action, as follows: In the Coflow multi-level feedback queue, from top to bottom, the lowest priority Q K To the highest priority Q1, use the method of dynamically changing the threshold to put the threshold of each priority queue into a set, that is, {TH 1, TH2,…,TH n }, where the priority level Q1 corresponds to a range from 0 to TH1, the priority level Q2 corresponds to a range from TH1 to TH2, and so on, until the lowest priority level Q K , the corresponding value range is from TH n To ∞; by the threshold {TH 1, TH2,…,TH n } to adjust the entire multi-level feedback queue; In a multi-level feedback queue, the newly arrived Coflow will first enter the highest priority queue. Then, as the size of the sent data increases, it will be downgraded according to the threshold of each priority, that is, each queue has a maximum threshold and a minimum threshold. That is, when the size of the sent data is greater than the first threshold TH1, the Coflow will be downgraded to the second priority queue. When the size of the sent data is greater than the second threshold TH2, the Coflow will be further downgraded to the third priority queue, and so on, until all the Coflow data is sent. Each priority has a different weight in network transmission. The higher the weight of the priority, the higher the network bandwidth allocated; the lower the weight of the priority, the lower the network bandwidth allocated. The threshold of the multi-level feedback queue can dynamically adjust the queue size of different priorities, thereby controlling data transmission.

7. The Coflow scheduling system based on the TD3 algorithm according to claim 3 is characterized in that: The environment state definition unit is used to define the environment state in Coflow-based deep reinforcement learning, and represents the size of Coflow, the number of flows included, the arrival time, and the duration as the state information of the environment in the form of a multidimensional array, as follows: ； Among them, each four elements represent the information of a Coflow, and then the sorting operation is performed based on the arrival time, and the arrival time is sorted from small to large; During operation, the order of entering the multi-level feedback queue is determined according to the sequence, and the queue where the Coflow is located is adjusted at the next moment; if the transmitted data is greater than the maximum threshold of the queue, the priority is lowered.

8. The Coflow scheduling system based on the TD3 algorithm according to claim 3, characterized in that: The reward function definition unit is used to define the reward function based on Coflow deep reinforcement learning and score the actions implemented in the current environment, as follows: Set the reward function formula: ； in, 、 is the continuous transmission time of the nth Coflow at time t and t+1; the difference between the continuous transmission time from time t+1 to time t is made, and the average of all N Coflows is calculated to reflect the change in the average completion time of the Coflow. Finally, the average value is normalized by the time difference to eliminate the influence of the time interval on the result; when a Coflow has completed the transmission, the latter term The result is 0.

9. The Coflow scheduling system based on the TD3 algorithm according to claim 4, characterized in that: The policy function is based on the actor-critic model, also known as the Actor-Critic model. The deep reinforcement learning algorithm adopted is the double-delayed deep deterministic policy gradient algorithm, also known as the TD3 algorithm. It includes six neural networks in total, two of which are actor neural networks, namely the actor network and the actor target network, and four are critic neural networks, namely critic network 1, critic network 2, critic target network 1, and critic target network 2. Among them, the Actor neural network is used to generate the corresponding action according to the state, and the Critic neural network is used to evaluate the action taken.

Images