TSN routing selection method based on DDQN

Abstract

The present application provides a time-sensitive network (TSN) routing selection method based on Dueling Deep Q Network (DDQN), the method comprising: based on a multi-controller software defined network (SDN), dividing the routing into multiple areas according to the industrial site, and constructing a system model of the TSN network; constructing a Markov decision model for the routing selection problem; obtaining topology and flow information, searching for reachable paths to generate a candidate path table; initializing the routing path for the service flow to form an initial network; using the average minimum end-to-end delay as the optimization target and the delay requirement of the flow as a constraint, using the DDQN algorithm to optimize and train the initial network; when the topology or service flow information changes, retraining the model; the controller assigns a routing path to the service flow and sends it down to the forwarding plane. This is a data-driven centralized multi-controller routing method that can be deployed in an industrial TSN network, can adapt to dynamically changing business and cross-workshop signal transmission requirements, and ensure the service quality of business transmission.

Description

Technical Field

[0001] The present invention relates to the technical field of TSN communication, and in particular to a TSN routing selection method based on DDQN. Background Art

[0002] Flexible manufacturing is a common practice in the field of industrial manufacturing. This model is consumer-oriented and is in contrast to the traditional large-scale mass production model. In flexible manufacturing, the response speed of the production line and supply chain is tested, and field equipment is generally present in different workshops or areas. It is necessary to have the ability to interconnect various business units and to communicate data between the field and the cloud at the same time. However, the satisfaction of real-time device interconnection and data communication requirements in this scenario is not high. It is now necessary to deploy the TSN network between the controller and the field equipment and between the controllers to achieve high-quality deterministic delay transmission of control signals and high-precision synchronous transmission of collaborative signals. At the same time, a suitable TSN routing algorithm is needed to improve the system's ability to adapt to external changes (such as changes in business flow caused by changes in production demand) and internal changes (such as equipment failures). At present, most routing selection schemes in the industrial manufacturing field are customized for traditional rigid automated production lines. It is assumed that the business flow information is fixed, and no targeted routing scheme is proposed for the typical network in which field equipment exists in different workshops. Therefore, it is impossible to provide the best routing path for the business flow. The network targeting and delay determinism are low, and the transmission service quality cannot be guaranteed. An adaptive intelligent routing selection method with network targeting and delay determinism is needed. Summary of the invention

[0003] In order to solve the low satisfaction of data intercommunication requirements and the defects of existing routing technologies when TSN is deployed in the industrial manufacturing field, an adaptive intelligent routing selection method that is sensitive to service quality is proposed. The optimization goal is to minimize the average end-to-end delay to ensure the transmission service quality of dynamically changing business flows in the flexible industrial manufacturing field. The steps are as follows:

[0004] Step 1: Build a multi-controller collaborative TSN network model based on SDN, divide the forwarding plane routing into multiple areas based on the workshop situation, build a mathematical model of the forwarding plane network topology with router ports as nodes, and jump to step 2;

[0005] Step 2: Construct a Markov decision model for the routing selection problem in the TSN network, determine the state space, action space, and reward function, and jump to step 3;

[0006] Step 3: Get the service flow information, search for all the reachable paths of the flow and put them into the candidate path table for standby, then jump to step 4;

[0007] Step 4: Based on the mathematical model of the forwarding plane network topology and the candidate path table, the existing TSN routing algorithm is used to initialize the routing path for the service flow to form an initialization network, and then jump to step 5;

[0008] Step 5: The controller, as an intelligent agent, takes the minimization of the average end-to-end delay as the optimization goal and the delay requirement of each service flow as the constraint, and optimizes and trains the initial network using the DDQN algorithm to obtain the optimal routing selection strategy for each service flow, and then jumps to step 6;

[0009] Step 6: The controller allocates a routing path for each service flow according to the optimal routing selection strategy, sends it to the forwarding plane, and jumps to step 7;

[0010] Step 7: Determine whether the topology or service flow information has changed. If so, jump to step 8; otherwise, jump to step 10.

[0011] Step 8: Determine whether the topology has changed or the service flow has increased. If so, jump to step 3; otherwise, jump to step 9.

[0012] Step 9: Update network resource usage information and jump to step 5;

[0013] Step 10: The routing policy remains unchanged.

[0014] Furthermore, the step 1 includes the following specific steps:

[0015] Step 1-1: The SDN-based multi-controller collaborative TSN network model includes multiple controllers of the control plane, the routing topology of the forwarding plane, and the application plane. The controller is divided into a centralized controller and multiple sub-controllers, which can obtain the routing topology information. Each sub-controller corresponds to a workshop, and the centralized controller is connected to all sub-controllers to control the global information.

[0016] Step 1-2: The workshop situation refers to the presence of field equipment in different workshops. The routing of the forwarding plane is divided into regions according to their characteristics, and one workshop is divided into one region.

[0017] Step 1-3: The mathematical model of the forwarding plane network topology is an ordered pair (V, E) based on regional division, denoted as graph G = (V, E), where V = {v 1 , v 2 , ..., v n}Define the set of router ports, E = {e 1 , e 2 , ..., e m} Define the set of links between router ports. The links are bidirectional. The distance between each point in the point set V is expressed using the distance matrix Indicates that w ij represents the distance between node i and node j, where

[0018] Furthermore, the step 2 includes the following specific steps:

[0019] Step 2-1: The TSN network is a forwarding plane network topology, which has been divided into regions and a mathematical model has been constructed;

[0020] Step 2-2: In the Markov decision process, at each discrete time step t, the agent observes the state s of the environment t , and performs an action according to its strategy, then receives a timely reward, and the environment enters the next state s t+1 , the conversion process is defined as a four-tuple t , a t , r t ,s t+1 > (3);

[0021] Step 2-3: s in the quaternary t ∈S represents the state of the environment at time t, which includes network topology information, router port occupancy, and current training service flow information;

[0022] Step 2-4: a in the quaternary t ∈A is the action at agent t, including the path for forwarding the currently trained data packet, i.e., the routing path it decides to forward the data packet;

[0023] Step 2-5: r in the quaternary t ∈R is the reward received by the agent when simulating sending a data packet through the routing path, which is defined as r t =-(p t +q t )(4), where p t Indicates that the data packet executes action a t The estimated queuing delay after t represents the propagation delay of the routing path;

[0024] Step 2-6: s in the quaternary t+1 ∈S means in state s t Execute action a t The next state after.

[0025] Furthermore, the step 3 includes the following specific steps:​

[0026] Step 3-1: The service flow information includes the flow period T k , Packet size M k , Deadline D k , Priority PR k Starting point PS k and endpoint PE k , where the subscript k is the flow mark, indicating the business flow f k ∈F flow information;

[0027] Step 3-2: The reachable path refers to searching for all reachable links from the starting point to the end point, and putting them into the candidate path table for standby, wherein each service flow has a corresponding candidate path table;

[0028] Step 3-3: The acquisition of topology and service flow information requires checking whether the topology has changed. If the topology has changed, re-search the reachable paths of all service flows and generate the corresponding candidate path table. If the topology has not changed, only the candidate path table of the newly added service flow is generated, and other service flows still use the original candidate path table.

[0029] Furthermore, the step 4 includes the following specific steps:

[0030] Step 4-1: Obtain service flow information and calculate the super period of the service flow, where the super period refers to the lowest common multiple of all service flow periods in the network, defined as T sup =LCM(T 1 , T 2 , ..., T 3 ) (5);

[0031] Step 4-2: In the current network environment t Under this circumstance, the existing TSN routing algorithm is used to allocate initial routing paths to the service flows in turn. After each flow path is allocated, the resource occupancy information of the simulated network is updated according to its routing and scheduling method until the initial path is allocated to all service flows.

[0032] Furthermore, the step 5 includes the following specific steps:

[0033] Step 5-1: Arrange all service flows in the network in descending order of priority to obtain a sequence table of flows to be trained;

[0034] Step 5-2: According to the sequence table, the service flows in the table are optimized and trained in sequence to obtain an optimized routing path, wherein before optimizing and training the current flow, the occupation of network resources by its initial routing strategy is first removed, and the network resource occupation information is updated, wherein the current flow refers to a service flow being trained;

[0035] Step 5-3: When optimizing the current flow, first determine whether its starting point and end point belong to the same area. If they belong to the same area, use the initial routing path as the optimized routing path; if they belong to different areas, use the sub-controller to calculate the routing path in the same area. When transmitting across areas, the central controller coordinates the sub-controller to use the DDQN algorithm to calculate the optimized routing path for the current flow;

[0036] Step 5-4: The DDQN algorithm is to minimize the end-to-end delay as the goal, and to use the DDQN algorithm to calculate the optimal path with the delay requirement of the flow and the training time not less than the super period as constraints. In DDQN, the Q network is modeled as: Q η,α，β (s, a) = V η，α (s)+A n,β (s, a)(6), where V η，α (s) is the state value function, A η，β (s, a) is the advantage function of taking different actions in this state, η is the network parameter shared by the state value function and the advantage value function, α and β are the parameters of the state value function and the advantage value function respectively;

[0037] Step 5-5: The stopping condition of the optimization training is that the average end-to-end delay of all service flows in the network tends to be stable, that is, reaches the minimum value. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the TSN network routing algorithm according to an embodiment of the present invention;

[0039] Figure 2 The control plane and forwarding plane models of the multi-controller SDN architecture of the embodiment of the present invention;

[0040] Figure 3 A schematic diagram of a network topology in which a network topology with routes as nodes is converted into a network topology in which ports are nodes according to an embodiment of the present invention (solid lines indicate that ports between routes are reachable, and dotted lines indicate that ports within routes are reachable);

[0041] Figure 4 Schematic diagram of a Markov decision model based on the DDQN algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0043] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprising" and similar words mean that the original or object appearing before the word covers the original or object listed after the word and its equivalent, without excluding other originals or objects.

[0044] As described in the background technology section, most routing selection schemes in the related art assume that the service flow information is fixed, and do not propose targeted routing schemes for typical networks in the field of flexible industrial manufacturing. In the process of implementing this application, the applicant found that the technical solutions in the related art have the following problems: they cannot provide the best routing path for dynamically changing service flows, the network targeting and delay certainty are low, and the service quality of service flow transmission cannot be guaranteed.

[0045] In view of this, the present application provides a TSN routing selection method based on DDQN, which uses DDQN to perform routing selection on the business flow of the TSN network in the industrial manufacturing field, and can provide the best routing path for dynamically changing business flows. The network has high targetedness and latency determinism, and can ensure its transmission service quality.

[0046] The technical solution of the present application is described in detail below through specific embodiments.

[0047] refer to Figure 1 An embodiment of the present application provides a TSN routing selection method based on DDQN, which specifically includes the following steps:

[0048] Step 1: Build a multi-controller collaborative TSN network model based on SDN, divide the forwarding plane routing into multiple areas according to the workshop situation, build a mathematical model of the forwarding plane network topology with router ports as nodes, and jump to step 2;

[0049] Step 2: Construct a Markov decision model for the routing selection problem in the TSN network, determine the state space, action space, and reward function, and jump to step 3;

[0050] Step 3: Obtain service flow information, search for all reachable paths and put them into the candidate path table for standby use, and then jump to step 4;

[0051] Step 4: The controller, as an intelligent agent, uses the DDQN algorithm to initialize the routing path for the service flow to form an initialization network based on the mathematical model of the forwarding plane network topology and the Markov decision model, with minimization of the end-to-end delay as the optimization goal and the service flow delay requirement as the constraint, and then jumps to step 5;

[0052] Step 5: The controller, as an intelligent agent, takes the minimization of the average end-to-end delay as the optimization goal and the delay requirement of each service flow as the constraint, and optimizes and trains the initial network using the DDQN algorithm to obtain the optimal routing selection strategy for each service flow, and then jumps to step 6;

[0053] Step 6: The controller allocates a routing path for each service flow according to the optimal routing selection strategy, sends it to the forwarding plane, and jumps to step 7;

[0054] Step 7: Determine whether the topology or service flow information has changed. If so, jump to step 8; otherwise, jump to step 10.

[0055] Step 8: Determine whether the topology has changed or the service flow has increased. If so, jump to step 3; otherwise, jump to step 9.

[0056] Step 9: Update network resource usage information and jump to step 5;

[0057] Step 10: The routing policy remains unchanged.

[0058] Furthermore, the step 1 includes the following specific steps:

[0059] Step 1-1: The SDN-based multi-controller collaborative TSN network model includes a forwarding plane, a control plane, and an application plane, wherein the control plane includes a centralized controller and multiple sub-controllers, and the forwarding plane includes a routing topology. Figure 2 , each sub-controller corresponds to a region, the centralized controller is connected to all sub-controllers, and can grasp the global information, and the controller obtains the routing topology information;

[0060] Step 1-2: The workshop situation refers to the presence of field equipment in different workshops. The routing of the forwarding plane is divided into regions according to their characteristics. Figure 2 , the same workshop is divided into one area;

[0061] Step 1-3: The mathematical model of the forwarding plane network topology is an ordered pair (V, E) based on regional division, denoted as graph G = (V, E), combined with Figure 3 , V = {v 1 , v 2 , ..., v n}Define the set of router ports, E = {e 1 , e 2 , ..., e m} Define the set of links between router ports. The links are bidirectional. The distance between each point in the point set V is expressed using the distance matrix Indicates that wij represents the distance between node i and node j, where

[0062] Furthermore, the step 2 includes the following specific steps:

[0063] Step 2-1: The TSN network is a forwarding plane network topology, which has been divided into regions and a mathematical model has been constructed;

[0064] Step 2-2: Combine Figure 4 , the Markov decision process observes the state s of the environment at each discrete time step t t , and performs an action a according to its strategy t , and then receives an immediate reward r t , the environment enters the next state s t+1 ;

[0065] Step 2-3: The t represents the state of the environment at time t, which includes network topology information, router port occupancy, and current training service flow information;

[0066] Step 2-4: t is the action at agent t, including the currently trained packet forwarding path decision, i.e., it determines the routing path for packet forwarding;

[0067] Step 2-5: The t It is the reward received by the agent after simulating sending a data packet through the routing path. Its value is the inverse of the delay of the routing path after executing the action, where the total delay is the sum of the queuing delay and the propagation delay.

[0068] Step 2-6: The S t+1 Indicates that in state s t Execute action a t The next state after.

[0069] Furthermore, the step 3 includes the following specific steps:

[0070] Step 3-1: The service flow information includes flow period, data packet size, deadline, priority starting point and end point;

[0071] Step 3-2: The reachable path refers to searching all reachable links from the starting point to the end point using existing technology (such as a depth-first search algorithm), and putting them into the candidate path table for standby, wherein each service flow has a corresponding candidate path table;

[0072] Step 3-3: The acquisition of topology and service flow information requires checking whether the topology has changed. If the topology has changed, re-search the reachable paths of all service flows and generate the corresponding candidate path table. If the topology has not changed, only the candidate path table of the newly added service flow is generated, and other service flows still use the original candidate path table.

[0073] Furthermore, the step 4 includes the following specific steps:

[0074] Step 4-1: Obtain service flow information, and calculate a super period for the service flow, wherein the super period refers to the least common multiple of all service flow periods in the network;

[0075] Step 4-2: In the current network environment t Under the condition that the end-to-end delay of the flow is constrained, the existing TSN routing algorithm is used to allocate routing paths to the service flows in turn. After each flow path is allocated, the resource occupancy information of the simulated network is updated according to its routing and scheduling method until the initial path is allocated for all service flows. Among them, the scheduling strategy can be any service flow scheduling algorithm in the prior art, such as the first-come-first-served algorithm (FCFS), the short job first algorithm (SJF), the high response ratio priority algorithm, the priority scheduling algorithm, etc.

[0076] Step 4-3: The stopping condition of network initialization is that all flows obtain the initialized routing path.

[0077] Furthermore, the step 5 includes the following specific steps:

[0078] Step 5-1: Arrange all service flows in the network in descending order of priority to obtain a sequence table of flows to be trained;

[0079] Step 5-2: According to the sequence table, the service flow in the table is cyclically optimized and trained to output the optimal action, that is, the optimized routing path is obtained. Before optimizing and training the current flow, the occupation of network resources by its initial routing strategy is first removed, and the current network environment s is updated. t , that is, updating the network resource occupancy information, wherein the current flow refers to a service flow being trained, and the cyclic optimization training means that if the stop condition of the optimization training is not met after the last flow in the sequence table is trained, the training will continue from the first flow;

[0080] Step 5-3: When optimizing the current flow, first determine whether its starting point and end point belong to the same area. If they belong to the same area, use the initialization routing path as the optimized routing path; if they belong to different areas, the sub-controller corresponding to the area in the same area calculates the routing path using the DDQN algorithm. When transmitting across areas, the central controller coordinates the sub-controller to use the DDQN algorithm to initialize the routing path;

[0081] Step 5-4: The DDQN algorithm is to minimize the end-to-end delay as the goal, with the delay requirement of the flow and the training time not less than the super period as the constraint, and use the DDQN algorithm to calculate the optimal path, combined with Figure 4 , in DDQN, the Q network is modeled as: Q η,α，β (s, a) = V η，α (s)+A η，β (s, a)(6), where V η，α (s) is the state value function, A η,β (s, a) is the advantage function of taking different actions in this state, η is the network parameter shared by the state value function and the advantage value function, α and β are the parameters of the state value function and the advantage value function respectively;

[0082] Step 5-5: The stopping condition of the optimization training is that the average end-to-end delay of all service flows in the network tends to be stable, that is, reaches the minimum value.

Claims

1. A TSN routing selection method based on DDQN, characterized in that: The following steps are involved: Build a multi-controller collaborative TSN network model based on SDN, divide the forwarding plane routing into multiple areas based on the workshop situation, and build a mathematical model of the forwarding plane network topology with router ports as nodes; Construct a Markov decision model for routing selection in TSN networks and determine the state space, action space, and reward function; Obtain business flow information, search for reachable paths for all flows and put them into the candidate path table for standby; Based on the mathematical model of the forwarding plane network topology and the candidate path table, the existing TSN routing algorithm is used to initialize the routing path for the service flow to form an initialization network; The controller includes: a centralized controller and multiple sub-controllers. The controller is used as an intelligent agent. Based on the mathematical model of the forwarding plane network topology and the Markov decision model, the initial network is optimized and trained using the DDQN algorithm with the minimization of the average end-to-end delay as the optimization goal and the delay requirements of each service flow as the constraint, and the optimal routing selection strategy for each service flow is obtained. The controller allocates a routing path for each service flow based on the optimal routing selection strategy and sends it to the forwarding plane; If the topology or service flow information changes, the network resource occupancy information is updated and the training is re-optimized; The process of optimizing and training the initial network using the DDQN algorithm includes: firstly arranging all the business flows in the network in descending order of priority to obtain a sequence table of the flow to be trained; secondly, according to the sequence table, optimizing and training the business flows in the table in turn to obtain an optimized routing path; wherein, before optimizing and training the current flow, first remove the occupation of network resources by its initial routing strategy, and update the network resource occupation information; wherein, the current flow refers to a business flow being trained; and finally, when optimizing and training the current flow, first determine whether its starting point and end point belong to the same area; if they belong to the same area, use its initialized routing path as the optimized routing path; if they belong to different areas, use the sub-controller in the same area to calculate the routing path; when transmitting across areas, the centralized controller coordinates the sub-controller to use the DDQN algorithm to calculate the optimized routing path for the current flow.

2. The TSN routing selection method based on DDQN according to claim 1, characterized in that: The SDN-based multi-controller collaborative TSN network model includes multiple controllers of the control plane, the routing topology of the forwarding plane, and the application plane. The controller is divided into a centralized controller and multiple sub-controllers, which can obtain the routing topology information. Each sub-controller corresponds to a workshop, and the centralized controller is connected to all sub-controllers to control the global information.

3. The DDQN-based TSN routing selection method according to claim 1, characterized in that: The workshop situation refers to that field equipment exists in different workshops, and the routing of the forwarding plane is divided into regions according to their characteristics, and one workshop is divided into one region.

4. The DDQN-based TSN routing selection method according to claim 1, characterized in that: The mathematical model of the forwarding plane network topology is an ordered pair (V, E) based on area division, denoted as graph G = (V, E), where V = {v1, v2, ..., v n }Define the set of router ports, E = {e1, e2, ..., e m } Define the set of links between router ports. The links are bidirectional. The distance between each point in the point set V is expressed using the distance matrix Indicates that w ij represents the distance between node i and node j, where 5. The TSN routing selection method based on DDQN according to claim 1, characterized in that: The Markov decision model for the routing selection problem in TSN networks includes: at each discrete time step t, the agent observes the state s of the environment t , and performs an action according to its strategy, then receives a timely reward, and the environment enters the next state s t+1 , the conversion process is defined as a four-tuple t ,a t ,r t ,s t+1 >(3), s in the quaternary t ∈S represents the state of the environment at t, which includes network topology information, router port occupancy and current training service flow information. t ∈A is the action of the agent t, including the path of the currently trained data packet forwarding, that is, the routing path it decides to forward the data packet. t ∈R is the reward received by the agent when simulating sending a data packet through the routing path, which is defined as r t =-(p t +q t )(4), where p t Indicates that the data packet executes action a t The estimated queuing delay after t Represents the propagation delay of the routing path, and s in the four-tuple t+1 ∈S means in state s t Execute action a t The next state after.​ 6. The DDQN-based TSN routing selection method according to claim 1, characterized in that: The service flow information includes the flow period T k , Packet size M k , Deadline D k , Priority PR k Starting point PS k and endpoint PE k , where the subscript k is the flow mark, indicating the service flow f k ∈F flow information.

7. The DDQN-based TSN routing selection method according to claim 1, characterized in that: A reachable path refers to searching for all reachable links from a starting point to an end point, and putting them into the candidate path table for standby use, wherein each service flow has a corresponding candidate path table.

8. The DDQN-based TSN routing selection method according to claim 1, characterized in that: To obtain topology and service flow information, it is necessary to check whether the topology has changed. If the topology has changed, the reachable paths of all service flows are searched again and the corresponding candidate path table is generated. If the topology has not changed, only the candidate path table of the newly added service flow is generated, and other service flows still use the original candidate path table.

9. The TSN routing selection method based on DDQN according to claim 1, characterized in that: The process of using the existing TSN routing algorithm to initialize the routing path for the service flow to form an initialization network includes: the current network environment s t Under this circumstance, the existing TSN routing algorithm is used to allocate initial routing paths to the business flows in turn. After each flow path is allocated, the resource occupancy information of the simulated network is updated according to its routing and scheduling method until the initial path is allocated to all business flows.

10. The TSN routing selection method based on DDQN according to claim 1, characterized in that: The DDQN algorithm aims to minimize the end-to-end delay, and takes the delay requirement of the flow and the training time as constraints. The optimal path is calculated using the DDQN algorithm. In DDQN, the Q network is modeled as: Q η,α,β (s,a)=V η,α (s)+A η,β (s,a)(6), where V η,α (s) is the state value function, A η,β (s,a) is the advantage function of taking different actions in this state, η is the network parameter shared by the state value function and the advantage value function, and α and β are the parameters of the state value function and the advantage value function respectively.

11. The TSN routing selection method based on DDQN according to claim 1, characterized in that: The stopping condition for the optimization training is that the average end-to-end delay of all service flows in the network tends to be stable, that is, reaches the minimum value.

Images