Service flow scheduling method, apparatus and device based on deep reinforcement learning, and medium

Abstract

Disclosed in the present application are a service flow scheduling method, apparatus and device based on deep reinforcement learning, and a medium. The method comprises: acquiring target data, wherein the target data comprises a current target topology state, current target service flow information and current target link information of a time-sensitive network; and inputting the target data into a deep reinforcement learning model to obtain a scheduling scheme, so as to implement scheduling of service flows on the basis of the scheduling scheme, wherein the deep reinforcement learning model comprises an agent and an environment, the agent is configured to determine gate control orchestration of the scheduling scheme, and the environment is configured to determine slot orchestration of the scheduling scheme on the basis of the gate control orchestration.

Description

Business flow scheduling method, device, equipment and medium based on deep reinforcement learning

[0001] Related applications

[0002] This application claims priority to Chinese patent application No. 202410310518.9 filed on March 18, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the technical field of business flow scheduling, and in particular to a business flow scheduling method, device, equipment and medium based on deep reinforcement learning. Background Art

[0004] Time-Sensitive Networking (TSN) proposes a gating mechanism called Time-Aware Shaper (TAS) in the IEEE 802.1Qbv standard. This mechanism enables fine-grained control of the transmission queue status in network switches through a pre-configured gate control list (GCL). However, in traditional TSN service flow scheduling algorithms, the length of the generated GCL list may be too large, exceeding the GCL length limit of existing network switches. This limitation has become a significant bottleneck in the actual deployment of scheduling solutions. To address this technical problem, related solutions have designed a flexible gating mechanism that mainly enables gating functions for specific service flows, thereby effectively reducing the length of the GCL list. However, this flexible gating mechanism still cannot achieve global service flow scheduling while meeting network transmission requirements.

[0005] Summary of the Invention

[0006] The main purpose of this application is to provide a business flow scheduling method, device, equipment and medium based on deep reinforcement learning, aiming to solve the technical problem that the relevant technology adopts the existing flexible gating mechanism and cannot realize the scheduling of global business flows while meeting network transmission requirements.

[0007] To achieve the above objectives, the present application provides a service flow scheduling method based on deep reinforcement learning, which includes the following steps:

[0008] Acquire target data, wherein the target data includes a current target topology state of the time-sensitive network, current target service flow information, and current target link information;

[0009] The target data is input into a deep reinforcement learning model to obtain a scheduling scheme to implement the scheduling of business flows based on the scheduling scheme, wherein the deep reinforcement learning model includes an agent and an environment, the agent is used to determine the gating arrangement of the scheduling scheme, and the environment is used to determine the time slot arrangement of the scheduling scheme based on the gating arrangement.

[0010] In one embodiment, before inputting the target data into the deep reinforcement learning model to obtain the scheduling solution, the method further includes:

[0011] Determine the transmission path of the current business flow under the current scheduling network;

[0012] Obtaining the current topology state, current service flow information, and current link information of the current scheduling network;

[0013] After the environment encodes the current topology state, the current service flow information, and the current link information into an observation state vector, the observation state vector is transmitted to the agent through the environment;

[0014] After the intelligent agent outputs the decision action of the current business flow on the transmission path according to the observation state vector, the allocation result of the environment allocating time slots to the current business flow according to the decision action is determined, the reward value is determined according to the allocation result, and the parameters of the initial deep reinforcement learning model are updated according to the reward value to obtain the deep reinforcement learning model.

[0015] In one embodiment, the agent includes a feature extractor and a policy network, the policy network includes an actor network and a critic network, the actor network includes a fully connected layer, and the critic network includes a fully connected layer; wherein, after the agent outputs a decision action of the current service flow on the transmission path according to the observed state vector, determining an allocation result of the environment allocating a time slot to the current service flow according to the decision action includes:

[0016] After the feature extractor converts the observed state vector into a feature vector, the decision probability of each action is determined according to the feature vector through the Actor network;

[0017] After the Critic network outputs a decision action for the current service flow on the transmission path according to the decision probability, an allocation result of the time slot allocated by the environment to the current service flow according to the decision action is determined.

[0018] In one embodiment, the observed state vector includes a first key-value pair, a second key-value pair, and a third key-value pair, the first key-value pair being used to represent the network topology state, the second key-value pair being used to represent the service flow information, and the third key-value pair being used to represent the link information. The feature extractor includes a graph neural network encoder and a feedforward neural network, the graph neural network encoder includes a graph isomorphism network, the graph isomorphism network includes a graph convolution layer and a global average pooling layer, and the feedforward neural network includes a linear layer and an activation function layer. After the feature extractor converts the observed state vector into a feature vector, the policy network outputs a decision action for the current service flow on the transmission path according to the feature vector, including:

[0019] After the graph neural network encoder encodes the first key-value pair into a first output vector and the feedforward neural network encodes the second key-value pair and the third key value into a second output vector, the first output vector and the second output vector are vector-concatenated to obtain the feature vector, and a decision action is output based on the feature vector through the policy network.

[0020] In one embodiment, determining an allocation result of the environment allocating a time slot to the current service flow according to the decision action, and determining a reward value according to the allocation result includes:

[0021] When the allocation result is that a legal time slot cannot be allocated to the current service flow, determining the reward value according to a fixed value;

[0022] When the allocation result successfully allocates a legal time slot for the current service flow, the reward value is determined based on a fixed reward, a first negative feedback, and a second negative feedback, wherein the first negative feedback is used to represent negative feedback on the gating list, and the second negative feedback is used to represent negative feedback on the time slot scheduling.

[0023] In one embodiment, determining an allocation result of the environment allocating a time slot to the current service flow according to the decision action, and determining a reward value according to the allocation result includes:

[0024] When the allocation result is that a legal time slot cannot be allocated to the current service flow, determining the reward value according to a fixed value;

[0025] When the allocation result successfully allocates a legal time slot for the current business flow, the gating list length, the maximum gating list length, and the interference time slot length are determined according to the current link situation, and the reward value is determined according to the gating list length, the maximum gating list length, the interference time slot length, and the end-to-end time slot requirement of the current business flow.

[0026] In one embodiment, updating the parameters of the initial deep reinforcement learning model according to the reward value to obtain the deep reinforcement learning model includes:

[0027] When the allocation result is that a legal time slot cannot be allocated to the current service flow, updating the parameters of the initial deep reinforcement learning model according to the reward value to obtain the deep reinforcement learning model; or

[0028] When the decision actions of all business flows are determined, the parameters of the initial deep reinforcement learning model are updated according to the reward value to obtain the deep reinforcement learning model.

[0029] In addition, to achieve the above objectives, the present application also proposes a service flow scheduling device based on deep reinforcement learning, which includes:

[0030] An acquisition module, configured to acquire target data, wherein the target data includes a current target topology state of the time-sensitive network, current target service flow information, and current target link details;

[0031] An input module is configured to input the target data into a deep reinforcement learning model to obtain a scheduling scheme to implement service flow scheduling based on the scheduling scheme, wherein the deep reinforcement learning model includes an agent and an environment, the agent is configured to determine a gating arrangement of the scheduling scheme, and the environment is configured to determine a time slot arrangement of the scheduling scheme based on the gating arrangement.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a business flow scheduling device based on deep reinforcement learning, and the business flow scheduling device based on deep reinforcement learning includes: a memory, a processor, and a business flow scheduling program based on deep reinforcement learning stored on the memory and runnable on the processor, and the business flow scheduling program based on deep reinforcement learning is configured to implement the steps of the business flow scheduling method based on deep reinforcement learning as described above.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, on which a business flow scheduling program based on deep reinforcement learning is stored. When the business flow scheduling program based on deep reinforcement learning is executed by a processor, the steps of the business flow scheduling method based on deep reinforcement learning as described above are implemented.

[0034] The service flow scheduling method, apparatus, device and medium based on deep reinforcement learning proposed in this application obtain target data, wherein the target data includes the current target topology state of the time-sensitive network, the current target service flow information and the current target link information; the target data is input into the deep reinforcement learning model to obtain a scheduling scheme to implement the scheduling of service flows based on the scheduling scheme, wherein the deep reinforcement learning model includes an intelligent agent and an environment, the intelligent agent is used to determine the gating arrangement of the scheduling scheme, and the environment is used to determine the time slot arrangement of the scheduling scheme according to the gating arrangement. In the above manner, the trained deep reinforcement learning model is used to process the scheduling problem of the network, and an effective scheduling arrangement scheme that meets the service quality requirements of the service flow can be generated, thereby realizing the scheduling of the global service flow while meeting the network transmission requirements, thereby ensuring the efficient and stable operation of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] FIG1 is a schematic diagram of the structure of a service flow scheduling device based on deep reinforcement learning in a hardware operating environment according to an embodiment of the present application;

[0036] FIG2 is a flow chart of a first embodiment of a method for scheduling service flows based on deep reinforcement learning in this application;

[0037] FIG3 is a flow chart of a second embodiment of a method for scheduling service flows based on deep reinforcement learning according to the present invention;

[0038] FIG4 is a diagram of a training framework based on a deep reinforcement learning model in a second embodiment of a service flow scheduling method based on deep reinforcement learning of the present application;

[0039] FIG5 is an architecture diagram of a deep reinforcement learning model agent according to a second embodiment of the service flow scheduling method based on deep reinforcement learning of the present application;

[0040] FIG6 is a diagram of an exemplary network topology structure in the second embodiment of the service flow scheduling method based on deep reinforcement learning of the present application;

[0041] FIG7 is a schematic diagram of time slot arrangement of a service flow in the second embodiment of the service flow scheduling method based on deep reinforcement learning of the present application;

[0042] FIG8 is a schematic diagram of time slot arrangement of two service flows in the second embodiment of the service flow scheduling method based on deep reinforcement learning of the present application;

[0043] Figure 9 is a structural block diagram of the first embodiment of the business flow scheduling device based on deep reinforcement learning in this application.

[0044] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0046] Refer to Figure 1, which is a schematic diagram of the structure of a business flow scheduling device based on deep reinforcement learning in the hardware operating environment involved in the embodiment of the present application.

[0047] As shown in Figure 1, the service flow scheduling device based on deep reinforcement learning may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0048] Those skilled in the art will understand that the structure shown in Figure 1 does not constitute a limitation on the business flow scheduling device based on deep reinforcement learning, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0049] As shown in FIG1 , the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a business flow scheduler based on deep reinforcement learning.

[0050] In the business flow scheduling device based on deep reinforcement learning shown in Figure 1, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the business flow scheduling device based on deep reinforcement learning of this application can be set in the business flow scheduling device based on deep reinforcement learning, and the business flow scheduling device based on deep reinforcement learning calls the business flow scheduling program based on deep reinforcement learning stored in the memory 1005 through the processor 1001, and executes the business flow scheduling method based on deep reinforcement learning provided in the embodiment of this application.

[0051] Based on the above hardware structure, an embodiment of a business flow scheduling method based on deep reinforcement learning is proposed in this application.

[0052] Refer to Figure 2, which is a flow chart of a first embodiment of a business flow scheduling method based on deep reinforcement learning in the present application.

[0053] In this embodiment, the service flow scheduling method based on deep reinforcement learning includes the following steps:

[0054] Step S10: Acquire target data, wherein the target data includes the current target topology state of the time-sensitive network, current target service flow information, and current target link information.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a mobile phone, tablet computer, personal computer, etc., or an electronic device capable of implementing the above functions or a service flow scheduling device based on deep reinforcement learning. The following uses the service flow scheduling device based on deep reinforcement learning as an example to illustrate this embodiment and the following embodiments.

[0056] It should be noted that the current target service flow state includes service flow period, transmission message size, end-to-end QoS requirements and QoS parameters (such as jitter requirements); the current target link state includes GCL state and time slot state.

[0057] Step S20: Input the target data into a deep reinforcement learning model to obtain a scheduling scheme to implement the scheduling of business flows based on the scheduling scheme, wherein the deep reinforcement learning model includes an agent and an environment, the agent is used to determine the gating arrangement of the scheduling scheme, and the environment is used to determine the time slot arrangement of the scheduling scheme based on the gating arrangement.

[0058] It should be noted that the deep reinforcement learning model refers to a trained deep reinforcement learning model, and the intelligent agent (i.e., agent) refers to an entity that can make decisions. For example, it can learn and optimize through the input target data to determine the gating orchestration of the scheduling plan, and the environment (i.e., Env.) will generate the time slot orchestration of the business flow based on the gating orchestration; gating orchestration refers to determining the decision-making actions of each real-time business flow on the link. For gating orchestration, the intelligent agent is responsible for determining the gating status of each business flow on each network node (i.e., whether gating is enabled) in each decision step. These decisions are implemented through the actions of each step; time slot orchestration refers to the reasonable arrangement and scheduling of different business flows on the timeline so that they can be effectively transmitted on the link without interfering with each other. For time slot orchestration, the environment is responsible for finding a legal time slot orchestration for the business flow given the decision-making actions of the intelligent agent.

[0059] In the specific implementation, after processing the target data, the deep reinforcement learning model can output a scheduling plan that meets the QoS requirements of the business flow to the network controller, which then configures the scheduling plan. Specifically, it is necessary to configure the starting sending time offset of each real-time business flow at its sending end, as well as the gate control list (GCL) in the switch in the network.

[0060] This embodiment obtains target data, wherein the target data includes the current target topology state of the time-sensitive network, the current target business flow information, and the current target link information; the target data is input into a deep reinforcement learning model to obtain a scheduling scheme to implement the scheduling of business flows based on the scheduling scheme, wherein the deep reinforcement learning model includes an agent and an environment, the agent is used to determine the gating arrangement of the scheduling scheme, and the environment is used to determine the time slot arrangement of the scheduling scheme according to the gating arrangement. In the above manner, by using a trained deep reinforcement learning model to process the scheduling problem of the network, an effective scheduling arrangement scheme that meets the service quality requirements of the business flow can be generated, and then the scheduling of the global business flow can be achieved while meeting the network transmission requirements, thereby ensuring the efficient and stable operation of the network.

[0061] Refer to Figure 3, which is a flow chart of a second embodiment of a business flow scheduling method based on deep reinforcement learning in the present application.

[0062] Based on the first embodiment described above, the service flow scheduling method based on deep reinforcement learning in this embodiment further includes:

[0063] Step S101: Determine the transmission path of the current service flow in the current scheduling network.

[0064] It should be noted that any path calculation method may be used to determine the transmission path of the current service flow in the current scheduling network, specifically, the Dijkstra algorithm may be used.

[0065] Step S102: Acquire the current topology state, current service flow information, and current link information of the current scheduling network.

[0066] It should be noted that the network topology of the time-sensitive network should be consistent with the network topology of the current scheduling network used in the training process. In order to ensure the performance of the algorithm, the characteristics of the business flow of the target data should be consistent with the characteristics of the business flow of the training process.

[0067] Step S103: After the environment encodes the current topology state, the current service flow information, and the current link information into an observation state vector, the observation state vector is transmitted to the agent through the environment.

[0068] It should be noted that the current service flow information includes the service flow period, transmission message size, end-to-end QoS requirements, and QoS parameters (such as jitter requirements); the current target link information includes the GCL status and time slot status. The current topology state, current service flow information, and electrical conceptual link information obtained by the environment are encoded into an observation state vector, where the observation state vector contains three parts of information, as follows:

[0069] (1) Network topology state: It consists of two matrices: the adjacency matrix and the feature vector matrix of each port node in the topology. Specifically, the node feature vector matrix is ​​a 2-dimensional N*M matrix, where N is the number of ports in the network topology and M is the number of features for each port. The features include the following port information: the number of cycle types of traffic flowing through the port, the time slot utilization of the link, the current GCL length, the current GCL period, and the maximum GCL length supported by the port.

[0070] (2) Characteristic information of the current real-time service flow: This is a high-dimensional vector that includes the service flow period, data volume, end-to-end delay QoS requirement, current hop count (hop index), accumulated jitter up to the current node, and jitter QoS requirement;

[0071] (3) The outgoing link status of the current flow at the current node: It is a high-dimensional vector that contains the link utilization, GCL period, GCL length, and statistical information of each service flow in each period.

[0072] In the specific implementation, taking Python programming as an example, the environment (Env.) encodes the observation state vector into a dictionary dict and outputs it to the agent (agent), which contains 3 key-value pairs corresponding to the above three parts of information.

[0073] Step S104: After the intelligent agent outputs the decision action of the current business flow on the transmission path according to the observation state vector, determine the allocation result of the environment allocating time slots to the current business flow according to the decision action, determine the reward value according to the allocation result, and update the parameters of the initial deep reinforcement learning model according to the reward value to obtain the deep reinforcement learning model.

[0074] It should be noted that the environment (Env.) allocates time slots for the current business flow based on the output decision action, and feeds back the reward value (reward) to the agent after calculating it, so as to update the parameters of the initial deep reinforcement learning model based on the interaction process between the agent and the environment.

[0075] In specific implementations, the initial deep reinforcement learning model can be trained through a variety of reinforcement learning algorithms, such as the typical PPO algorithm.

[0076] It is understandable that in order to solve the complex scheduling problem currently faced in time-sensitive networks (TSN), the calculation process of this complex scheduling problem involves the optimization of the gating list and the scheduling of time slots to ensure that all business flows can meet their specific network transmission requirements, such as end-to-end delay and jitter. As shown in Figure 4, through the deep reinforcement learning model, the agent (Agent) determines the gating action (i.e., decision-making action) of the current business flow on the current link based on the state it observes, which contains various information in the network as of the current scheduling state (including network topology status, current business flow information, current link details, etc.). Then the environment (Env.) will evaluate the legitimacy of this action, calculate the corresponding reward, and update the state accordingly, and iteratively schedule the next node or the next business flow.

[0077] In one embodiment, the agent includes a feature extractor and a policy network, the policy network includes an actor network and a critic network, the actor network includes a fully connected layer, and the critic network includes a fully connected layer; wherein, after the agent outputs a decision action of the current service flow on the transmission path according to the observed state vector, determining an allocation result of the environment allocating a time slot to the current service flow according to the decision action includes:

[0078] After the feature extractor converts the observed state vector into a feature vector, the decision probability of each action is determined according to the feature vector through the Actor network;

[0079] After the Critic network outputs a decision action for the current service flow on the transmission path according to the decision probability, an allocation result of the time slot allocated by the environment to the current service flow according to the decision action is determined.

[0080] It should be noted that, as shown in Figure 5, the intelligent agent includes a feature extractor and a policy network. The feature extractor is responsible for converting the observed state vector into a feature vector, while the policy network is responsible for outputting the decision action based on the feature vector mapping; the policy network includes an Actor network and a Critic network. The Actor network consists of two fully connected layers, and the Critic network consists of two fully connected layers. The actor network outputs the decision probability of each action and performs sampling. The Critic network then outputs the decision action of the current business flow on the transmission path based on the decision probability.

[0081] In one embodiment, the observed state vector includes a first key-value pair, a second key-value pair, and a third key-value pair, the first key-value pair being used to represent the network topology state, the second key-value pair being used to represent the service flow information, and the third key-value pair being used to represent the link information. The feature extractor includes a graph neural network encoder and a feedforward neural network, the graph neural network encoder includes a graph isomorphism network, the graph isomorphism network includes a graph convolution layer and a global average pooling layer, and the feedforward neural network includes a linear layer and an activation function layer. After the feature extractor converts the observed state vector into a feature vector, the policy network outputs a decision action for the current service flow on the transmission path according to the feature vector, including:

[0082] After the graph neural network encoder encodes the first key-value pair into a first output vector and the feedforward neural network encodes the second key-value pair and the third key value into a second output vector, the first output vector and the second output vector are vector-concatenated to obtain the feature vector, and a decision action is output based on the feature vector through the policy network.

[0083] It should be noted that the feature extractor includes a graph neural network encoder and a feedforward neural network, wherein the graph neural network encoder is used to process the first key-value pair, and the feedforward neural network is used to process the second key-value pair and the third key-value pair. Specifically, the graph neural network encoder is a graph isomorphism network (GIN) model. The graph isomorphism model structure mainly includes two layers of graph convolution layers GINConv and a global average pooling layer (Global_mean_pool). The global average pooling layer averages the features of each node in the output graph of the previous layer to obtain a vector representing the network, as shown below:

[0084] Where N is the number of ports in the network topology, and It is the vector output of each port obtained from the output of the GINConv operation in the previous layer.

[0085] It should be noted that this graph neural network encoding can aggregate the feature information of adjacent nodes into each node. Since the scheduling process of business flows involves the orchestration of each node on the forwarding path in the network, the use of graph isomorphic networks can better represent the features of each node in the network topology, thereby training a neural network that is better suited to solving orchestration tasks.

[0086] It should be noted that the feedforward neural network includes a linear layer and a ReLU activation function.

[0087] The output vector of the graph neural network (i.e., the first output vector) and the output vector of the feedforward neural network (i.e., the second output vector) are then concatenated to obtain a feature vector, which is then input into the policy network. Finally, the policy network is used to obtain the decision action in the current state, i.e., whether to enable gating for the current business flow at the current node.

[0088] In the specific implementation, using Python as an example, the open-source neural network library PyTorch, the graph neural network library PyG, and the reinforcement learning library Stable-Baselines3 (SB3) were used for encoding. The feature extractor input is the dict-type state information output by Env., encoded as a 128-dimensional vector. The GIN encoder outputs a 64-dimensional vector, while the feedforward neural network outputs another 64-dimensional vector. The GIN encoder first uses the GINConv graph convolution layer from the PyG library, then uses the BatchNorm1d batch norm layer from the PyTorch library for normalization. The GINConv graph convolution layer and BatchNorm1d batch norm layer are then repeated. Finally, global mean pooling is performed using the global_mean_pool function from the PyG library to output a 64-dimensional vector. This is then concatenated with the 64-dimensional vector output by the feedforward neural network that processes the link state and service flow state to form a 128-dimensional vector embedding.

[0089] For the policy network, the PPO network in SB3 is selected, and its default network structure and parameters can be used, that is, a 2-layer fully connected network with 64 units in each layer, a learning rate of 0.0003, an Adam optimizer, etc.

[0090] In one embodiment, determining an allocation result of the environment allocating a time slot to the current service flow according to the decision action, and determining a reward value according to the allocation result includes:

[0091] When the allocation result is that a legal time slot cannot be allocated to the current service flow, determining the reward value according to a fixed value;

[0092] When the allocation result successfully allocates a legal time slot for the current service flow, the reward value is determined based on a fixed reward, a first negative feedback, and a second negative feedback, wherein the first negative feedback is used to represent negative feedback on the gating list, and the second negative feedback is used to represent negative feedback on the time slot scheduling.

[0093] In the specific implementation, the environment (Env.) allocates time slots for the current business flow based on the output decision action, and feeds back the reward value to the agent after calculating the reward value. The specific calculation method of the reward value is as follows:

[0094] It can be understood that if the environment cannot allocate a legal time slot for the current business flow, a negative feedback (i.e., a fixed value) is returned. If Env. cannot allocate a legal time slot for the current business flow, the current episode (i.e., the interaction between the current environment and the agent) is terminated. If the environment successfully allocates a legal time slot for the current business flow, its reward should be a fixed reward A. However, in order to better train the agent's perception of the gated list GCL and time slot scheduling, the reward value can also include negative feedback B (i.e., the first negative feedback) on the impact of GCL and negative feedback C (i.e., the second negative feedback) on the time slot scheduling.

[0095] In one embodiment, determining an allocation result of the environment allocating a time slot to the current service flow according to the decision action, and determining a reward value according to the allocation result includes:

[0096] When the allocation result is that a legal time slot cannot be allocated to the current service flow, determining the reward value according to a fixed value;

[0097] When the allocation result successfully allocates a legal time slot for the current business flow, the gating list length, the maximum gating list length, and the interference time slot length are determined according to the current link situation, and the reward value is determined according to the gating list length, the maximum gating list length, the interference time slot length, and the end-to-end time slot requirement of the current business flow.

[0098] In the specific implementation, the fixed value is set to -1. When the allocation result is that a legal time slot cannot be allocated to the current service flow, the returned reward value is -1. When the allocation result is that a legal time slot is successfully allocated to the current service flow, the reward value can be determined based on the gating list length, the maximum gating list length, the interference time slot length, and the end-to-end time slot requirement of the current service flow. The specific calculation method is as follows:

[0099] Where g([u,t] t ,s t ) represents the current link [u,t] tAs of now, the length of gcl, and G([u,v] t ) represents the maximum length of the link, t IF It represents the interference time slot length of the link, and fd represents the end-to-end time slot requirement of the current service flow.

[0100] In one embodiment, updating the parameters of the initial deep reinforcement learning model according to the reward value to obtain the deep reinforcement learning model includes:

[0101] When the allocation result is that a legal time slot cannot be allocated to the current service flow, updating the parameters of the initial deep reinforcement learning model according to the reward value to obtain the deep reinforcement learning model; or

[0102] When the decision actions of all business flows are determined, the parameters of the initial deep reinforcement learning model are updated according to the reward value to obtain the deep reinforcement learning model.

[0103] It should be noted that if all business flows have completed scheduling (that is, the decision actions of all business flows have been determined), or the environment cannot allocate a legal time slot for the current business flow (due to insufficient network time slot resources), the episode (that is, the interaction between the agent and the environment) is terminated, and the parameters of the initial deep reinforcement learning model are updated based on the reward value generated by the interaction process between the agent and the environment; otherwise, the observation state vector needs to be updated and the scheduling of the next business flow is iteratively performed until the cumulative number of interactions between the agent and the environment is greater than the preset threshold, the training is terminated, and the deep reinforcement learning model is obtained.

[0104] In the specific implementation, as shown in Figure 6, the network contains two real-time service flows, both sending nodes are Node A and receiving nodes are Node E. The forwarding path is (ABCDE). In each episode (interaction process) of training the agent, service flow 1 is first scheduled at the first node Node A, then at the second node Node B, and finally at Node E. Figure 7 shows a feasible time slot scheduling result for service flow 1. In this example, the agent only enables gating at Node C. Enabling gating at Node C can bring the following benefits:

[0105] 1. Eliminate jitter accumulated at preceding nodes and reduce end-to-end jitter: During the forwarding process of this service flow, the preceding node, Node B, does not have gating enabled, which introduces additional jitter that accumulates hop by hop. Enabling gating at Node C effectively eliminates the jitter accumulated at preceding nodes. In large networks, such as those with long service flow forwarding paths, accumulated jitter can be significant. In such cases, enabling gating at intermediate nodes can eliminate this accumulated jitter, effectively reducing end-to-end jitter.

[0106] 2. Reduce timeslot resources and improve port utilization: As shown in Figure 7, when gating is not enabled on the leading node, the additional jitter introduced accumulates hop by hop, resulting in a larger reserved timeslot and reducing port utilization. Enabling gating on the intermediate node can effectively reduce the size of the reserved timeslot, thereby improving port utilization.

[0107] It is understandable that enabling gating also has a side effect, namely increasing the number of GCL entries on the port. When the GCL period is inconsistent with the period of the current flow, enabling gating will greatly increase the number of GCL entries. For example, when the port GCL period is 128ms and the service flow period is 1ms, enabling gating will add 256 GCL entries, which is difficult to deploy for actual TSN switches with limited GCL capabilities. Therefore, when making gating decisions, the above-mentioned deep neural network needs to consider not only the gain brought by enabling gating, but also its side effects. Specifically, these two constraints can be expressed in the reward feedback calculation formula through the first negative feedback and the second negative feedback, respectively.

[0108] In the specific implementation, the agent schedules service flow 2 only after it completes scheduling for service flow 1. As shown in Figure 8, a feasible time slot scheduling scheme for the two service flows is demonstrated. In this example, the agent enables gating for service flow 2 at Node B and Node D, and the time slot scheduling of the environment (Env.) ensures that there is no time slot conflict between the two service flows. This scheduling scheme enables gating for only one service flow at Node B / C / D, and only two GCL entries are required to complete the scheduling of real-time service flows. This can significantly reduce the GCL resource occupation of the network while ensuring the real-time transmission of network service flows.

[0109] This embodiment determines the transmission path of the current business flow under the current scheduling network; obtains the current topological state, current business flow information and current link information of the current scheduling network; after the environment encodes the current topological state, the current business flow information and the current link information into an observation state vector, the observation state vector is transmitted to the agent through the environment; after the agent outputs the decision action of the current business flow on the transmission path according to the observation state vector, the allocation result of the environment allocating time slots for the current business flow according to the decision action is determined, the reward value is determined according to the allocation result, and the parameters of the initial deep reinforcement learning model are updated according to the reward value to obtain the deep reinforcement learning model. Through the above method, the entire scheduling problem is decomposed into a series of serialized decision sub-problems, so that in each interaction process during the training process, the real-time business flow to be scheduled is scheduled by the agent in a flow-by-flow and hop-by-hop manner, thereby achieving effective solution through reinforcement learning.

[0110] In addition, an embodiment of the present application also proposes a storage medium, on which a business flow scheduling program based on deep reinforcement learning is stored. When the business flow scheduling program based on deep reinforcement learning is executed by a processor, the steps of the business flow scheduling method based on deep reinforcement learning as described above are implemented.

[0111] Refer to Figure 9, which is a structural block diagram of the first embodiment of the business flow scheduling device based on deep reinforcement learning in this application.

[0112] As shown in FIG9 , the service flow scheduling device based on deep reinforcement learning proposed in the embodiment of the present application includes:

[0113] The acquisition module 10 is configured to acquire target data, wherein the target data includes the current target topology state of the time-sensitive network, current target service flow information, and current target link details.

[0114] The input module 20 is used to input the target data into the deep reinforcement learning model to obtain a scheduling scheme to implement the scheduling of business flows based on the scheduling scheme, wherein the deep reinforcement learning model includes an agent and an environment, the agent is used to determine the gating arrangement of the scheduling scheme, and the environment is used to determine the time slot arrangement of the scheduling scheme based on the gating arrangement.

[0115] It should be understood that the above is only an example and does not constitute any limitation to the technical solution of the present application. In specific applications, technicians in this field can make settings as needed, and the present application does not impose any restrictions on this.

[0116] This embodiment obtains target data, wherein the target data includes the current target topology state of the time-sensitive network, the current target business flow information, and the current target link information; the target data is input into a deep reinforcement learning model to obtain a scheduling scheme to implement the scheduling of business flows based on the scheduling scheme, wherein the deep reinforcement learning model includes an agent and an environment, the agent is used to determine the gating arrangement of the scheduling scheme, and the environment is used to determine the time slot arrangement of the scheduling scheme according to the gating arrangement. In the above manner, by using a trained deep reinforcement learning model to process the scheduling problem of the network, an effective scheduling arrangement scheme that meets the service quality requirements of the business flow can be generated, and then the scheduling of the global business flow can be achieved while meeting the network transmission requirements, thereby ensuring the efficient and stable operation of the network.

[0117] In one embodiment, the input module 20 is further configured to:

[0118] Determine the transmission path of the current business flow under the current scheduling network;

[0119] Obtaining the current topology state, current service flow information, and current link information of the current scheduling network;

[0120] After the environment encodes the current topology state, the current service flow information, and the current link information into an observation state vector, the observation state vector is transmitted to the agent through the environment;

[0121] After the intelligent agent outputs the decision action of the current business flow on the transmission path according to the observation state vector, the allocation result of the environment allocating time slots to the current business flow according to the decision action is determined, the reward value is determined according to the allocation result, and the parameters of the initial deep reinforcement learning model are updated according to the reward value to obtain the deep reinforcement learning model.

[0122] In one embodiment, the agent includes a feature extractor and a policy network, the policy network includes an actor network and a critic network, the actor network includes a fully connected layer, and the critic network includes a fully connected layer; wherein the input module 20 is further used to:

[0123] After the feature extractor converts the observed state vector into a feature vector, the decision probability of each action is determined according to the feature vector through the Actor network;

[0124] After the Critic network outputs a decision action for the current service flow on the transmission path according to the decision probability, an allocation result of the time slot allocated by the environment to the current service flow according to the decision action is determined.

[0125] In one embodiment, the observed state vector includes a first key-value pair, a second key-value pair, and a third key-value pair, wherein the first key-value pair is used to represent the network topology state, the second key-value pair is used to represent the service flow information, and the third key-value pair is used to represent the link information. The feature extractor includes a graph neural network encoder and a feedforward neural network, the graph neural network encoder includes a graph isomorphism network, the graph isomorphism network includes a graph convolution layer and a global average pooling layer, and the feedforward neural network includes a linear layer and an activation function layer; wherein the input module 20 is further used to:

[0126] After the graph neural network encoder encodes the first key-value pair into a first output vector and the feedforward neural network encodes the second key-value pair and the third key value into a second output vector, the first output vector and the second output vector are vector-concatenated to obtain the feature vector, and a decision action is output based on the feature vector through the policy network.

[0127] In one embodiment, the input module 20 is further configured to:

[0128] When the allocation result is that a legal time slot cannot be allocated to the current service flow, determining the reward value according to a fixed value;

[0129] When the allocation result successfully allocates a legal time slot for the current service flow, the reward value is determined based on a fixed reward, a first negative feedback, and a second negative feedback, wherein the first negative feedback is used to represent negative feedback on the gating list, and the second negative feedback is used to represent negative feedback on the time slot scheduling.

[0130] In one embodiment, the input module 20 is further configured to:

[0131] When the allocation result is that a legal time slot cannot be allocated to the current service flow, determining the reward value according to a fixed value;

[0132] When the allocation result successfully allocates a legal time slot for the current business flow, the gating list length, the maximum gating list length, and the interference time slot length are determined according to the current link situation, and the reward value is determined according to the gating list length, the maximum gating list length, the interference time slot length, and the end-to-end time slot requirement of the current business flow.

[0133] In one embodiment, the input module 20 is further configured to:

[0134] When the allocation result is that a legal time slot cannot be allocated to the current service flow, updating the parameters of the initial deep reinforcement learning model according to the reward value to obtain the deep reinforcement learning model; or

[0135] When the decision actions of all business flows are determined, the parameters of the initial deep reinforcement learning model are updated according to the reward value to obtain the deep reinforcement learning model.

[0136] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In actual applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of this embodiment scheme, and no restrictions are imposed here.

[0137] In addition, for technical details not fully described in this embodiment, please refer to the business flow scheduling method based on deep reinforcement learning provided in any embodiment of the present application, and will not be repeated here.

[0138] In addition, it should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0139] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0140] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, or of course by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0141] The above are merely optional embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A service flow scheduling method based on deep reinforcement learning, wherein: The service flow scheduling method based on deep reinforcement learning includes: Acquire target data, wherein the target data includes a current target topology state of the time-sensitive network, current target service flow information, and current target link information; The target data is input into a deep reinforcement learning model to obtain a scheduling scheme to implement the scheduling of business flows based on the scheduling scheme, wherein the deep reinforcement learning model includes an agent and an environment, the agent is used to determine the gating arrangement of the scheduling scheme, and the environment is used to determine the time slot arrangement of the scheduling scheme based on the gating arrangement.

2. The method according to claim 1, wherein Before inputting the target data into the deep reinforcement learning model to obtain the scheduling solution, the method further includes: Determine the transmission path of the current business flow under the current scheduling network; Obtaining the current topology state, current service flow information, and current link information of the current scheduling network; After the environment encodes the current topology state, the current service flow information, and the current link information into an observation state vector, the observation state vector is transmitted to the agent through the environment; After the intelligent agent outputs the decision action of the current business flow on the transmission path according to the observation state vector, the allocation result of the environment allocating time slots to the current business flow according to the decision action is determined, the reward value is determined according to the allocation result, and the parameters of the initial deep reinforcement learning model are updated according to the reward value to obtain the deep reinforcement learning model.

3. The method according to claim 2, wherein: The intelligent agent includes a feature extractor and a policy network, the policy network includes an actor network and a critic network, the actor network includes a fully connected layer, and the critic network includes a fully connected layer; wherein, after the intelligent agent outputs a decision action of the current service flow on the transmission path according to the observed state vector, determining an allocation result of the environment allocating a time slot to the current service flow according to the decision action includes: After the feature extractor converts the observed state vector into a feature vector, the decision probability of each action is determined according to the feature vector through the Actor network; After the Critic network outputs a decision action for the current service flow on the transmission path according to the decision probability, an allocation result of the time slot allocated by the environment to the current service flow according to the decision action is determined.

4. The method according to claim 3, wherein: The observed state vector includes a first key-value pair, a second key-value pair, and a third key-value pair, wherein the first key-value pair is used to represent the network topology state, the second key-value pair is used to represent the service flow information, and the third key-value pair is used to represent the link information. The feature extractor includes a graph neural network encoder and a feedforward neural network, the graph neural network encoder includes a graph isomorphism network, the graph isomorphism network includes a graph convolution layer and a global average pooling layer, and the feedforward neural network includes a linear layer and an activation function layer. After the feature extractor converts the observed state vector into a feature vector, the policy network outputs a decision action for the current service flow on the transmission path according to the feature vector, including: After the graph neural network encoder encodes the first key-value pair into a first output vector and the feedforward neural network encodes the second key-value pair and the third key value into a second output vector, the first output vector and the second output vector are vector-concatenated to obtain the feature vector, and a decision action is output based on the feature vector through the policy network.

5. The method according to claim 2, wherein: The determining of an allocation result of the environment allocating a time slot to the current service flow according to the decision action, and determining a reward value according to the allocation result, includes: When the allocation result is that a legal time slot cannot be allocated to the current service flow, determining the reward value according to a fixed value; When the allocation result successfully allocates a legal time slot for the current service flow, the reward value is determined based on a fixed reward, a first negative feedback, and a second negative feedback, wherein the first negative feedback is used to represent negative feedback on the gating list, and the second negative feedback is used to represent negative feedback on the time slot scheduling.

6. The method of claim 2, wherein: The determining of an allocation result of the environment allocating a time slot to the current service flow according to the decision action, and determining a reward value according to the allocation result, includes: When the allocation result is that a legal time slot cannot be allocated to the current service flow, determining the reward value according to a fixed value; When the allocation result successfully allocates a legal time slot for the current business flow, the gating list length, the maximum gating list length, and the interference time slot length are determined according to the current link situation, and the reward value is determined according to the gating list length, the maximum gating list length, the interference time slot length, and the end-to-end time slot requirement of the current business flow.

7. The method of claim 2, wherein: The updating of the parameters of the initial deep reinforcement learning model according to the reward value to obtain the deep reinforcement learning model includes: When the allocation result is that a legal time slot cannot be allocated to the current service flow, updating the parameters of the initial deep reinforcement learning model according to the reward value to obtain the deep reinforcement learning model; or When the decision actions of all business flows are determined, the parameters of the initial deep reinforcement learning model are updated according to the reward value to obtain the deep reinforcement learning model.

8. A service flow scheduling device based on deep reinforcement learning, wherein: The service flow scheduling device based on deep reinforcement learning includes: An acquisition module, configured to acquire target data, wherein the target data includes a current target topology state of the time-sensitive network, current target service flow information, and current target link details; An input module is configured to input the target data into a deep reinforcement learning model to obtain a scheduling scheme to implement service flow scheduling based on the scheduling scheme, wherein the deep reinforcement learning model includes an agent and an environment, the agent is configured to determine a gating arrangement of the scheduling scheme, and the environment is configured to determine a time slot arrangement of the scheduling scheme based on the gating arrangement.

9. A service flow scheduling device based on deep reinforcement learning, wherein: The device includes: a memory, a processor, and a deep reinforcement learning-based business flow scheduler stored in the memory and executable on the processor, wherein the deep reinforcement learning-based business flow scheduler is configured to implement the steps of the deep reinforcement learning-based business flow scheduling method as described in any one of claims 1 to 7.

10. A storage medium, wherein: The storage medium stores a business flow scheduling program based on deep reinforcement learning. When the business flow scheduling program based on deep reinforcement learning is executed by the processor, the steps of the business flow scheduling method based on deep reinforcement learning as described in any one of claims 1 to 7 are implemented.

Images