A TSSDN dynamic routing decision method based on a DDPG deep reinforcement learning algorithm

Abstract

This invention proposes a dynamic routing decision-making method, specifically a TSSDN dynamic routing decision-making method based on the DDPG deep reinforcement learning algorithm. The aim is to design dynamic routing decisions using deep reinforcement learning algorithms based on network dynamic awareness prediction results, thereby reducing the transmission latency of low-priority flows in time-sensitive networks. This invention uses a deep learning algorithm to predict switch queue lengths in real time, and then makes next-hop routing decisions based on the prediction results. Higher prediction accuracy leads to better dynamic routing performance and improved decision-making efficiency. The implementation steps are: 1) Constructing the TSSDN network node architecture; 2) Constructing the network topology; 3) Constructing a topology feature extraction method based on PCA; 4) Constructing a prediction model based on a deep learning algorithm; 5) Constructing a routing decision-making model based on a deep reinforcement learning algorithm; 6) Iteratively training the deep reinforcement learning-based routing decision-making model based on the prediction results. This invention can be applied to scenarios such as telemedicine.

Description

Technical Field

[0001] This invention belongs to the field of computer network technology. It designs a dynamic routing decision method based on deep reinforcement learning, specifically a TSDDN dynamic routing decision method based on the DDPG deep reinforcement learning algorithm, which can be used in fields such as telemedicine and vehicular Ethernet. Background Technology

[0002] Software-defined Time-Sensitive Networking (TSSDN) architecture establishes and controls the TSN process by implementing four main management functions: monitoring, perception, decision-making, and execution, through an SDN controller. For TSNs, ultra-low latency data transmission is a critical requirement. The purpose of routing decisions is to achieve load balancing of data flows across network nodes, thereby reducing routing path latency. Routing decisions define the specific forwarding path of a flow, and the effectiveness of the strategy depends on the accuracy of perception and prediction of network conditions during transmission, especially the queue length of switch buffers. When a flow is transmitted in the network, if the queue length of the next-hop switch buffer cannot be perceived and accurately predicted in real time, then in the face of network congestion, if the data flow cannot avoid congested sections in time, its transmission latency cannot be guaranteed. Routing decision-making methods can be divided into traditional routing decision-making methods and deep reinforcement learning-based routing decision-making methods. Traditional routing decision-making methods lack perception and prediction of network switch conditions, easily causing network congestion and increasing data flow queuing latency. Deep reinforcement learning-based routing decision-making has a high predictive ability for changes in data traffic patterns in the network. When network congestion is predicted, the algorithm can adjust the path in time to avoid congested switches, thereby reducing queuing delay.

[0003] For example, patent application CN 110611619A, entitled "An Intelligent Routing Decision-Making Method Based on DDPG Reinforcement Learning Algorithm," incorporates a state gain algorithm to calculate the network sampling states between two adjacent iterations. This data is then input into a reinforcement learning algorithm, which outputs the required link information. The designed reward function is the maximum difference in network bandwidth utilization. Reinforcement learning continuously interacts with the network model, attempting and adjusting routing decisions. This method achieves network load balancing by adjusting data forwarding paths within the network. However, a drawback is that the SDN controller can only perform network awareness and cannot predict the future state of TSN switches based on traffic patterns. It cannot anticipate network congestion and involves extensive trial and error during decision-making, resulting in low algorithm efficiency and increased transmission latency for data streams in time-sensitive networks. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of the existing technologies by proposing a dynamic routing decision method based on deep reinforcement learning algorithms, which solves the problem of large data transmission latency in time-sensitive networks under high traffic conditions.

[0005] The technical approach of this invention is as follows: First, the switch status in the network topology is monitored in real time to collect the switch buffer queue length. Furthermore, this invention incorporates an LSTM-based switch queue length prediction algorithm, using the network-aware prediction results as features input into a deep reinforcement learning algorithm. Through multiple training iterations, a suitable algorithm model is obtained. Specifically, the following steps are included:

[0006] A dynamic routing decision method for TSDDN based on the DDPG deep reinforcement learning algorithm first predicts the queue length of switches in the time-sensitive network, and then makes routing decisions based on the queue prediction results, including the following steps:

[0007] S1. Construct the network topology;

[0008] S2. Extract network topology features;

[0009] S3. The SDN controller senses the TSN network status and determines whether it has changed. If so, return to step S2 to rebuild the topology; otherwise, proceed to step S4.

[0010] S4. Construct a switch queue length prediction model based on the LSTM deep learning algorithm of the Long Short-Term Memory network. Use the monitoring information monitored by the SDN controller as input to optimize and train the model so that it can be used to output the switch queue length.

[0011] S5. Construct a routing decision model based on the DDPG deep reinforcement learning algorithm;

[0012] S6. Using the predicted results of the perceived TSN network state as input, repeatedly train the routing decision model based on deep reinforcement learning, and output the routing decision model when the deadline for iteration is reached; the routing decision model is used to make routing decisions on the online real-time collected data stream, output the decision results, and the relevant TSN nodes in the network topology forward the data stream to the next hop node according to the routing decision results.

[0013] The network topology is as follows: a network topology graph G with N TSN nodes and degree d, where N≥10 and d≥2. Each TSN switch node is connected to other switch nodes through d edges to form the topology graph.

[0014] The extraction of network topology features includes:

[0015] (3a) Initialize the adjacency matrix X of the network topology. N×N ;

[0016] (3b) The covariance matrix of the network topology is calculated based on the principal component analysis (PCA) algorithm. The principal component score P of each switch is calculated as a network topology feature to represent the connection relationship between the switches.

[0017] The monitoring information monitored by the SDN controller includes: topology characteristics, traffic patterns, and switch queue length.

[0018] The construction of the switch queue length prediction model based on the Long Short-Term Memory (LSTM) deep learning algorithm includes:

[0019] (4a) Input and output: Topology features, traffic patterns and queue length status are used as inputs, and the TSN switch queue length in the next time slot is used as the output of the neural network.

[0020] (4b) Model Training: The TSN switch queue prediction model consists of L layers and t_x time steps. The L layers include input x, output y, and (L-2) hidden layers in the middle. Data propagates through the model via a forward formula, and the final output is the predicted value Q of the switch queue length. k ;

[0021] (4c) Model solution: The mean squared error (MSE) is selected as the objective function of the neural network. The gradient descent is solved by calculating the MSE value. Finally, the values ​​of each network parameter are obtained through learning. The Adam algorithm is then used for optimization.

[0022] The activation functions of the hidden layer and the output layer are respectively and

[0023] The construction of the routing decision model based on the DDPG deep reinforcement learning algorithm includes:

[0024] A routing decision model based on the deep deterministic policy gradient (DDPG) deep reinforcement learning algorithm is constructed. In the DDPG deep reinforcement learning algorithm, the action network and its target network in the behavior decision algorithm π, and the value network and its target network in the behavior value algorithm Q, are all neural networks with a three-layer fully connected network structure. The action network parameters θ and its target network parameters θ'←θ in π, and the value network parameters ω and its target network parameters ω'←ω in Q are randomly initialized and used to perceive the TSN network state and put it into the experience playback set D.

[0025] The routing decision model based on deep reinforcement learning is iteratively trained as follows:

[0026] (6a) Let the current iteration number be k, and the maximum iteration number be K, where K≥10 4 The initial sampling state of network topology G is S0, and k = 1, S0 = 0;

[0027] (6b) Obtain the current sampling state S of the time-sensitive network G. k =(Q k Delay k ); where Q k The queue prediction value Q output by the queue length prediction model k Delay k For the transmission delay of network data streams;

[0028] (6c) S k Transform into eigenvector φ(S) k ), and φ(S) k As input to the deep reinforcement learning-based routing decision model, the output routing decision behavior vector A is calculated. k This gives the next-hop switch address of the data stream.

[0029] (6d) Update G's execution decision A k The state S after TSN k ', and take it as the sampled state of G after the state transition, and calculate S. k 'Reward value R k At the same time, S k 'Transform into eigenvector φ(S) k ');

[0030] (6e) Establish the experience replay set D: {φ(S k ),A k ,R k ,φ(S k The function `is_end}` is used to transition the state of network topology G, where `is_end` indicates whether the state is terminated.

[0031] (6f) According to the sampling probability P(j), sample M samples from D, and calculate the loss function value of the behavior value algorithm and the loss function value of the behavior decision algorithm. Then, the value network parameters ω are updated using the loss statement, and simultaneously... Update the action network parameters θ, then add random noise to the updated θ and ω respectively. After several iterations, use the updated θ to update the target network parameters θ' of the action network, and use the updated ω to update the target network parameters ω' of the action network, where j represents the j-th sample, and M≥1;

[0032] (6g) via routing decision vector Ak The behavior value corresponding to each path is used to perceive and predict the status of the switch queue length in the TSN network.

[0033] (6h) Determine whether the cutoff condition has been reached. If so, obtain the dynamic routing policy in the TSN network. Otherwise, return to step (6b) to update the sampling state and continue iterating.

[0034] The reward value R mentioned in step (6d) k The calculation method is to add one to the negative of the data stream transmission delay.

[0035] A TSSDN dynamic routing decision controller based on the DDPG deep reinforcement learning algorithm, the controller comprising: a processor and a storage medium, the storage medium storing the following TSSDN network node architecture program modules: a queue module (Queue), a routing module (Routing), a flow table pattern module (App), and a decision module (Controller); when the processor loads and executes the program modules, the Controller calls the Queue, Routing, and App to execute the steps of the method described in any one of claims 1-9, thereby realizing the prediction of the switch queue length in the time-sensitive network and the output of routing decisions;

[0036] The App is used to simulate the generation of data streams in the network and store the data streams in a Queue for forwarding.

[0037] The Controller is used to obtain network information and perform queue prediction and routing decision steps, and to send the routing table to Routing;

[0038] The routing is used to receive the routing table output by the Controller and to forward the data stream in the Queue according to the next-hop address in the routing table.

[0039] The Queue is used to store the length of the switch queue. The data flow in the queue will be forwarded to the next hop node according to the routing table issued by Routing.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] Before making routing decisions based on deep reinforcement learning algorithms, this invention performs overall perception and prediction of the Time-Sensitive Network (TSN). By iteratively training based on changing data traffic patterns and network topology, it trains a deep learning-based switch queue prediction model with good prediction performance to predict changes in network state. Compared with existing technologies, this invention improves the efficiency of network routing decisions by predicting network state in advance and feeding the prediction results into the deep reinforcement learning-based dynamic routing decision model.

[0042] Because this invention extracts features from the network topology, it reduces the dimensionality of the representation of the network topology in the prediction algorithm, extracts the key connection information of the network topology, improves the training efficiency of the prediction algorithm, avoids the "curse of dimensionality" compared with the prior art, accelerates the convergence of the prediction algorithm, and improves the training speed of the prediction model.

[0043] Because this invention considers both network topology and network traffic patterns during network model prediction training, it avoids incomplete consideration of influencing factors that could affect prediction accuracy. Compared with existing technologies, it improves the input-output structure of the LSTM-based prediction model, enhances the prediction accuracy of time series, and lays a good foundation for subsequent routing decisions.

[0044] Because this invention incorporates the predicted queue length of switches in the network and the transmission delay of data streams in the network into the experience replay set when making network routing decisions, and designs the reward value as a function of network transmission delay, compared with the prior art, the routing decision effect is directly reflected by the data stream transmission delay, avoiding a large number of trial and error, improving the overall performance and accuracy of the algorithm, and further enhancing the ability to reduce network data transmission delay. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0046] Figure 2 This is a flowchart illustrating the implementation of iterative training of a deep reinforcement learning-based routing decision model according to the present invention.

[0047] Figure 3 This is a diagram of the TSDDN network node architecture of the present invention.

[0048] Figure 4 This is a structural diagram of the LSTM-based prediction model of the present invention. Detailed Implementation

[0049] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0050] This invention aims to design dynamic routing decisions using deep reinforcement learning algorithms based on network dynamic sensing and prediction results. This improves the ability of Time-Sensitive Networks (TSNs) to handle and adapt to sudden traffic changes, achieving network load balancing and thus reducing the transmission latency of low-priority data flows and increasing network throughput. The invention employs an LSTM-based deep learning prediction algorithm to predict switch queue lengths in real time, and then makes next-hop routing decisions based on the prediction results. Higher prediction accuracy leads to higher routing policy scores, better dynamic routing performance, and improved decision-making efficiency. Furthermore, adding noise to the neural network parameters facilitates system exploration and improves algorithm performance.

[0051] Reference Figure 1 The implementation steps of this invention are as follows:

[0052] Step 1), construct the TSSDN network node architecture:

[0053] Construct a network node model comprising four functionalities: Queue, Routing, App, and Controller. The Queue forwards messages, Routing generates a routing table and forwards messages, the App generates messages and receives messages at the application layer, and the Controller obtains network information and distributes flow tables. Refer to [reference needed] for the specific structure. Figure 3 .

[0054] Step 2), construct the network topology:

[0055] Construct a network topology graph G with N TSN nodes and degree d, where N≥10 and d≥2. Each TSN switch node is connected to other switch nodes through d edges to form the topology graph. In this example, a topology graph with 14 TSN switch nodes and degree 3 is selected, where N=14 and d=3.

[0056] Step 3) Construct a network topology feature extraction method based on Principal Component Analysis (PCA) algorithm:

[0057] (3a) Initialize the adjacency matrix X of the network topology. N×N Its mathematical expression is

[0058]

[0059] Where, x NN Indicates the pairwise connection relationship between nodes;

[0060] (3b) Calculate the covariance matrix of the network topology based on the Principal Component Analysis (PCA) algorithm. The formula for calculating the covariance is:

[0061]

[0062] Among them, X * X represents the normalized adjacency matrix. N×N Each element in the middle;

[0063] All principal components are linear combinations of the input variables, and each sample has a corresponding principal component score P, the mathematical expression of which is as follows:

[0064]

[0065] Among them, l NN Feature vectors; using the principal component scores of each switch to represent the connectivity of the switches can effectively reduce the dimensionality of the network adjacency matrix. At the same time, the scores can also reflect the connection relationships of the switches in the underlying network.

[0066] Step 4) Construct a switch queue length prediction model based on the Long Short-Term Memory (LSTM) deep learning algorithm.

[0067] (4a) Input / Output Design: Network Topology Representation, P ＜t＞ =[p ＜1＞ ,p ＜2＞ ,…,p ＜N＞ This indicates the connectivity of the switch, p ＜i＞ This indicates the switch's score within the network topology; the traffic pattern is the rate of packet growth on each TSN switch. This indicates the number of normalized data packets that arrived at the TSN switch in the previous time interval; Let m = 8N, where m represents the length of the switch queue lines in the network, N represents the number of switch queues, and t_x is the time interval. The topology, traffic pattern, and queue length status are used as inputs. The neural network should process the TSN switch queue length in the next time slot. Therefore, the LSTM prediction model structure is as follows: Figure 4 The mathematical expressions for the input and output are as follows:

[0068]

[0069]

[0070] (4b) Model Training: The TSN switch queue prediction model consists of L layers and t_x time steps. The L layers include input x, output y, and (L-2) hidden layers in the middle. In this example, L = 3, t_x = 10000, and the activation functions of the hidden and output layers are respectively... and When the input is a vector, the activation functions tanh(x) and σ(x) are also vectors. Furthermore, since σ(x) ∈ [0, 1], the output value is also normalized to [0, 1]. Data propagates through the model using the forward propagation formula, which is mathematically expressed as:

[0071]

[0072]

[0073] Among them, a ＜t＞ This represents the activation output value of the LSTM neural network in the current time slot; Indicates the predicted output value; W aa W ax b a W represents the activation parameters to be trained in the neural network. ya b y This represents the output parameters to be trained in the neural network.

[0074] (4c) Model Solving: The Mean Squared Error (MSE) is chosen as the objective function of the neural network. Gradient descent is performed by calculating the MSE value, and finally, the parameter values ​​in the Long Short-Term Memory (LSTM) neural network are obtained through learning. The LSM neural network is optimized using Adam. The mathematical expression for MSE is:

[0075]

[0076] in, This represents the predicted output value obtained through neural network prediction operations; y ＜t＞ t_x represents the actual value; N represents the number of nodes in the network topology; t_x represents the time step of the operation.

[0077] The neural network is optimized by Adam, a stochastic gradient-based optimization.

[0078] Step 5), construct a routing decision model based on the DDPG deep reinforcement learning algorithm:

[0079] A routing decision model based on the Deep Deterministic Policy Gradient (DDPG) deep reinforcement learning algorithm is constructed. In the DDPG deep reinforcement learning algorithm, the action network and its target network in the behavior decision algorithm π, and the value network and its target network in the behavior value algorithm Q, are all neural networks with a three-layer fully connected network structure. The action network parameters θ and its target network parameters θ'←θ in π, and the value network parameters ω and its target network parameters ω'←ω in Q are randomly initialized. The perceived TSN network state is placed in the experience replay set D. In this example, the experience replay set stores the network state and the transmission delay of the data stream in the network over a period of time. The length of D is 10000.

[0080] Step 6) Iteratively train the routing decision model based on deep reinforcement learning:

[0081] (6a) Let the number of iterations be k, and the maximum number of iterations be K, where K ≥ 10 4 The initial sampling state of network topology G is S0, and k = 1, S0 = 0;

[0082] See attached document Figure 2 The specific steps for training the routing decision model will be described in further detail.

[0083] (6b) The SDN controller senses the TSN network status and feeds the monitored information on network topology, traffic patterns, and switch queue lengths into the queue length prediction model for prediction. Simultaneously, it measures the network transmission latency of different types of data flows and calculates the predicted switch queue length Q. k and network data stream transmission delay k The current sampling state of G is denoted as S. k =(Q k Delay k );

[0084] (6c) S k Transform into eigenvector φ(S) k ), and φ(S) k As input to the deep reinforcement learning-based routing decision model, the output routing decision behavior vector A is calculated. k That is, the next-hop switch address of the data flow; the formula for calculating the decision behavior vector is: A = π θ (φ(S))+N, where N is a constant;

[0085] (6d) Obtain the execution decision A of G according to the method in step (6b). k The state S after TSN k ', and take it as the sampled state of G after the state transition, and then according to S k'Calculate the reward value R' k At the same time, S k 'Transform into eigenvector φ(S) k '), Reward value R k The calculation method is to add one to the negative of the data stream transmission delay, so the smaller the transmission delay, the larger the reward value;

[0086] (6e) will φ(S) k A k R k ,φ(S k '), is_end is combined to form empirical information {φ(S k ),A k ,R k ,φ(S k The process is as follows: '), is_end}, where is_end indicates whether it is a termination state, and it is stored in the experience replay set D to realize the transition of the network topology G state;

[0087] (6f) Arrange the samples in descending order of sampling probability P(j), and sample the M samples {φ(S)} with the highest probability values ​​from D. k ),A k ,R k ,φ(S k '),is_end}, and calculate the loss function value of the behavior value algorithm and the loss function value of the behavior decision algorithm using M samples. Then, the value network parameters ω are updated using the loss statement, and simultaneously... The action network parameters θ are updated, and then the updated θ and ω are added to random noise respectively. After every 10 iterations, the target network parameters θ' of the action network are updated using the updated θ, and the target network parameters ω' of the value network are updated using the updated ω, where j represents the j-th sample, M≥1, the sampling probability P(j), the loss function value of the behavior value algorithm, and the loss function value of the behavior decision algorithm are all represented. The calculation formula is:

[0088]

[0089]

[0090]

[0091] Among them, S j Let A represent the state of the j-th learning process. j p represents the action taken in the j-th learning process. j p represents the probability identifier. j =|δ j |+ε,δj δ represents the system evaluation error value. j =y j -Q(φ(S J ),A j ,ω),y j Indicates state S' j Value assessment quantity, y j =R j +γQ(φ(S′ j ),θ',ω'),R j Let represent the reward value of the j-th learning process, Q represent the behavior value algorithm, π represent the behavior decision algorithm, θ represent the action network parameters, ω represent the value network parameters, θ' represent the target network parameters of the action network, ω' represent the target network parameters of the value network, and γ represent the discount factor, γ = 0.9. j This represents the weight of the loss function for the j-th sample. min(P(j)) represents the minimum sampling probability P(j) among M samples. The random variable ε is a small random value, generally taken as a random number in the range of 0 to 0.3. α is a constant of 2, and β is a constant of 0.4. Let α represent the probability value of the current j-th sample raised to the power of α. This represents the summation of the probability values ​​of the currently selected M samples to the power of α. The updated θ updates the target network parameters θ' of the action network, and the updated ω updates the target network parameters ω' of the value network. The update methods are as follows:

[0092] ω'←τω+(1-τ)ω'

[0093] θ'←τθ+(1-τ)θ'

[0094] Where τ is the learning efficiency, τ = 0.5, and in this example M = 1000;

[0095] (6g) via routing decision vector A k The behavior value corresponding to each path is used to perceive and predict the status of the switch queue length in the TSN network.

[0096] (6h) Determine whether k = K is true. If yes, obtain the dynamic routing policy in the TSN network. Otherwise, let k = k + 1 and execute step (6b). In this example, K = 10000.

[0097] The embodiments described above will help those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

Claims

1. A TSSDN dynamic routing decision method based on the DDPG deep reinforcement learning algorithm, characterized in that, First, the queue length of the switches in the time-sensitive network is predicted. Then, routing decisions are made based on the queue prediction results, including the following steps: S1. Construct the network topology; the network topology is as follows: There are 1 TSN node with a degree of 1 Network topology diagram ,in, , Each TSN switch node passes through The edges are connected to other switch nodes to form a topology graph; S2. Extract network topology features; S3. The SDN controller senses the TSN network status and determines whether it has changed. If so, return to step S2 to rebuild the topology; otherwise, proceed to step S4. S4. Construct a switch queue length prediction model based on the LSTM deep learning algorithm of the Long Short-Term Memory network. Use the monitoring information monitored by the SDN controller as input to optimize and train the model so that it can be used to output the switch queue length. S5. Construct a routing decision model based on the DDPG deep reinforcement learning algorithm; S6. Using the predicted results of the perceived TSN network state as input, repeatedly train the routing decision model based on deep reinforcement learning, and output the routing decision model when the deadline for iteration is reached; the routing decision model is used to make routing decisions on the online real-time collected data stream and output the decision results, and the relevant TSN nodes in the network topology forward the data stream to the next hop node according to the routing decision results.

2. The TSSDN dynamic routing decision method based on the DDPG deep reinforcement learning algorithm according to claim 1, characterized in that, The extraction of network topology features includes: (3a) Initialize the adjacency matrix of the network topology ; (3b) Calculate the covariance matrix of the network topology based on the principal component analysis (PCA) algorithm, and calculate the principal component score of each switch. As a network topology feature, it is used to represent the connection relationships between switches.

3. The TSSDN dynamic routing decision method based on the DDPG deep reinforcement learning algorithm according to claim 1, characterized in that, The monitoring information monitored by the SDN controller includes: topology characteristics, traffic patterns, and switch queue length.

4. The TSSDN dynamic routing decision method based on the DDPG deep reinforcement learning algorithm according to claim 1, characterized in that, The construction of the switch queue length prediction model based on the Long Short-Term Memory (LSTM) deep learning algorithm includes: (4a) Input and output: Topology features, traffic patterns and queue length status are used as inputs, and the TSN switch queue length in the next time slot is used as the output of the neural network; (4b) Model Training: The TSN switch queue prediction model consists of L layers and t_x time steps. The L layers include input x, output y, and (L-2) hidden layers in the middle. Data propagates in the model through a forward formula, and finally outputs the predicted value of the switch queue length. ; (4c) Model solution: The mean squared error (MSE) is selected as the objective function of the neural network. The gradient descent is solved by calculating the MSE value. Finally, the values ​​of each network parameter are obtained through learning. The Adam algorithm is then used for optimization.

5. The TSSDN dynamic routing decision method based on the DDPG deep reinforcement learning algorithm according to claim 4, characterized in that, The activation functions of the hidden layer and the output layer are respectively and .

6. The TSSDN dynamic routing decision method based on the DDPG deep reinforcement learning algorithm according to claim 1, characterized in that, The construction of the routing decision model based on the DDPG deep reinforcement learning algorithm includes: Construct a routing decision model based on the Deep Deterministic Policy Gradient (DDPG) deep reinforcement learning algorithm, where the behavior decision algorithm in the DDPG deep reinforcement learning algorithm... The action network and its target network in the algorithm, as well as the value network and its target network in the behavior-value algorithm Q, both employ a three-layer fully connected neural network structure, with random initialization. Action network parameters and its target network parameters , and the value network parameters in Q and its target network parameters The data is used to sense the state of the TSN network and put it into the experience replay set D.

7. The TSSDN dynamic routing decision method based on the DDPG deep reinforcement learning algorithm according to claim 1, characterized in that, The routing decision model based on deep reinforcement learning is iteratively trained as follows: (6a) Let the current iteration number be k, and the maximum iteration number be K. The initial sampling state of network topology G is: And let k=1, ; (6b) Obtain the current sampling state of the time-sensitive network G. ;in, Queue length prediction model output queue prediction value , For the transmission delay of network data streams; (6c) will Convert to feature vector and will As input to the deep reinforcement learning-based routing decision model, the output routing decision behavior vector is calculated. This gives the next-hop switch address of the data stream. (6d) Update G's execution decision The state after TSN And take it as the sampled state of G after the state transition, and calculate Reward value At the same time Convert to feature vector ; (6e) Establish the experience replay set D: ,in Indicates whether it is a terminated state, and realizes the transition of network topology G state; (6f) According to sampling probability Sample M samples from D, and calculate the loss function value of the behavior value algorithm and the loss function value of the behavior decision algorithm. Then, the value network parameters are adjusted using the loss statement. Update, and simultaneously through Action network parameters Update it, and then... and Random noise is added to each iteration, and after a certain number of iterations, the updated noise is used. For the target network parameters of the action network Update and use the updated version. For the target network parameters of the action network Perform an update, where j represents the j-th sample. ; (6g) via routing decision vector The behavior value corresponding to each path is used to perceive and predict the status of the switch queue length in the TSN network. (6h) Determine whether the cutoff condition has been reached. If so, obtain the dynamic routing policy in the TSN network. Otherwise, return to step (6b) to update the sampling state and continue iterating.

8. The TSSDN dynamic routing decision method based on the DDPG deep reinforcement learning algorithm as described in claim 1, characterized in that, The reward value described in step (6d) The calculation method is to add one to the negative of the data stream transmission delay.

9. A TSSDN dynamic routing decision controller based on the DDPG deep reinforcement learning algorithm, characterized in that, The controller includes a processor and a storage medium, wherein the storage medium stores the following TSDDN network node architecture program modules: a queue module (Queue), a routing module (Routing), a flow table mode module (App), and a decision module (Controller); when the processor loads and executes the program modules, the Controller calls the Queue, Routing, and App to perform the steps of the method described in any one of claims 1-8, thereby predicting the switch queue length for time-sensitive networks and outputting routing decisions. The App is used to simulate the generation of data streams in the network and store the data streams in a Queue for forwarding. The Controller is used to obtain network information and perform queue prediction and routing decision steps, and to send the routing table to Routing; The routing is used to receive the routing table output by the Controller and to forward the data stream in the Queue according to the next-hop address in the routing table. The Queue is used to store the length of the switch queue. The data flow in the queue will be forwarded to the next hop node according to the routing table issued by Routing.

Images