A digital-twin-based SDN routing optimization method

Abstract

The application provides an SDN routing optimization method based on digital twinning, which comprises two spaces of a physical space and a virtual space, wherein a data plane composed of hosts and switches of an SDN and a control plane composed of controllers exist in the physical space, and a digital twin constructed by using hosts, switches and link information in the data plane exists in the virtual space; in the physical space, the control plane is responsible for completing network traffic forwarding control, an agent for executing a DT-DRL routing algorithm is constructed by the controller, the agent is denoted as DRL-Agent, and the DRL-Agent provides a routing algorithm calculation model and network state information based on digital twinning, and plans a suitable forwarding path for each piece of network traffic. The application can provide multi-dimensional network data and a self-adaptive updated routing strategy for the controller by digital twinning sensing, analyzing network environment data and optimizing a routing algorithm.

Description

Technical Field

[0001] This invention belongs to the field of software-defined networking and intelligent communication networks, and specifically relates to an SDN routing optimization method based on digital twins. Background Technology

[0002] With the development of Software-Defined Networking (SDN) technology, its application areas are constantly expanding. The number of user devices in SDN is rapidly increasing, and business needs are becoming more diverse, leading to increasingly diverse and complex network environments and communication requirements for SDN. Traditional SDN routing algorithms, due to limited information supporting decision-making and inflexible routing policy adjustment mechanisms, are increasingly unable to guarantee good routing transmission quality in complex and changing network environments. Existing routing research for SDN can be mainly divided into three categories: those based on traditional mathematical models, heuristic algorithms, and deep reinforcement learning. However, these routing algorithms all have certain limitations when facing complex and ever-changing network environments and communication requirements.

[0003] (1) Routing algorithms based on traditional mathematical models mainly allocate paths for traffic based on the current link status in the network. Most of these algorithms are based on the shortest path algorithm, calculating new link weights by weighting parameters such as link latency, packet loss rate, and queue length, and then using Dijkstra's algorithm to obtain the transmission path. This approach easily leads to link congestion from the same starting switch to the same ending switch, ultimately resulting in high packet loss rates and high latency. Furthermore, these routing algorithms typically respond according to pre-defined rules and cannot quickly adapt to dynamic changes in the network environment.

[0004] (2) Routing algorithms based on heuristics can be classified into ant colony optimization, genetic algorithm, and particle swarm optimization, etc., according to their solution principles. These algorithms can obtain the optimal solution that satisfies the constraints through continuous iterative calculations, thereby obtaining the optimal transmission path for traffic in the current network environment. However, heuristic algorithms have a long iteration process, which leads to a long computation delay in routing decisions and affects the efficiency of the controller's routing decisions.

[0005] (3) In DRL-based routing algorithms, the controller, as an intelligent agent, can gradually perceive and adapt to the dynamic changes in the network environment by constantly interacting with the network, and thus allocate the best transmission path for traffic in a complex and ever-changing network environment. However, the need for a large amount of historical network data for model training will put a lot of computational and communication pressure on the controller.

[0006] Digital twins, by constructing real-time digital mirrors of physical entities, enable the monitoring, simulation, and control of these entities in virtual space. Currently, digital twins are widely used in communication networks, such as for compute offloading decisions and edge cache placement. Introducing digital twins into SDN not only helps controllers perceive network environment information more broadly and granularly, but also assists in modeling and analyzing network data, providing a more comprehensive global network view. This enhances the systemic simulation, optimization, verification, and control capabilities of the network, contributing to better network management and maintenance, reducing network trial-and-error costs, and promoting network technology innovation.

[0007] In complex and ever-changing SDN networks, traditional routing algorithms, due to their limited decision-making basis and low flexibility, are unable to plan suitable paths for each traffic flow, leading to link congestion, load imbalance, and other issues, resulting in high packet loss rates and communication latency. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a digital twin-based SDN routing optimization method that not only provides the controller with multi-dimensional network data and adaptively updates routing policies by sensing and analyzing network environment data and optimizing routing algorithms through digital twins, but also reduces the communication and computing pressure on the controller to optimize routing algorithms.

[0009] The objective of this invention is achieved through the following technical solution: an SDN routing optimization method based on digital twins, comprising two spaces: physical space and virtual space. The data plane, composed of SDN hosts and switches, and the control plane, composed of controllers, exist in the physical space. The digital twin, constructed using host, switch, and link information from the data plane, exists in the virtual space. In the physical space, the control plane is responsible for network traffic forwarding control. The controller constructs an agent to execute the DT-DRL routing algorithm, denoted as DRL-Agent. The DRL-Agent, based on the routing algorithm calculation model and network state information provided by the digital twin, plans a suitable forwarding path for each network traffic item.

[0010] The DT-DRL routing algorithm includes a Policy Network deployed in the controller, a Training Network deployed in the digital twin, and a Target Network for auxiliary training. The digital twin analyzes network status and collects decision-making experience based on historical network data mapped from the data plane, and updates the experience replay pool of the routing algorithm based on the decision-making experience. The routing algorithm is dynamically updated by periodically training the Training Network and updating the parameters of the Policy Network to the controller. When the controller receives a routing request, it uses traffic information and information provided by the digital twin as the environmental status, selects the optimal path for the traffic from multiple alternative paths, and calculates the routing table of the relevant switches based on the path to complete the efficient transmission of traffic.

[0011] Suppose the data plane has C switches and N links. The data plane can be simplified to an undirected graph G = {SW, E}, where SW = {s1, s2, ..., s}. C} represents a set of switches, where s is a set of switches. i Let H be the set of connected hosts. i ={h i，1 h i，2 ,...}, i∈[1, C]; E={e 1，2 e 1，3 , ..., e 2，3 , ...} represents a set consisting of N links, e i,j s i and s j There exists a link, i∈[1,C], j∈[1,C], i≠j; it has a communication distance d i，j Link bandwidth b i，j and link utilization u i,j (t) Three attributes, where d i，j and b i，j u is a constant. i,j (t) represents e i,j The time that data passes through the link per unit of time, and this variable changes with time t.

[0012] Source host h i，j to the destination host h i′，j′At time t, a flow request is initiated. First, the flow request is transmitted to the switch connected to the source host. The flow contains three attributes for routing decisions: source host address, destination host address, and flow size, where i ≠ i′ and j ≠ j′. When the switch in the data plane receives the flow, it reports it to the controller. The controller then uses the network status and flow information provided by the digital twin as the status input of the PolicyNetwork. The action decision output by the PolicyNetwork will serve as the optimal transmission path for the flow. After calculation by the routing table, it is sent to the relevant switches in the network, thus completing the transmission of the flow.

[0013] Once a flow is transmitted to the destination host, when mapping the host's operational status data to the digital twin next time, the digital twin can determine a reward value for the DRL-Agent based on the actual packet loss rate and latency of that flow, and will then store the network state space information s, effective decision information a, and reward value r for that decision. t The network state space information s′ at the next moment is used as a decision experience and put into the experience replay pool; the packet loss rate loss(t) is represented by the number of packets Num that reach the destination host in this flow. R Number of packets sent by the source host (Num) T The ratio, i.e.

[0014]

[0015] When the amount of experience in the experience replay pool meets the preset conditions, the Training Network and Target Network in the digital twin will learn the historical network state changes through experience replay, adjust the neuron parameters in the Training Network and Target Network, and synchronize the parameters in the Training Network with the parameters in the Policy Network in the DRL-Agent to achieve asynchronous updates of the routing algorithm.

[0016] The beneficial effects of this invention are:

[0017] 1. This invention proposes an SDN routing optimization technology architecture based on digital twins. By using digital twins to sense and analyze network environment data and optimize routing algorithms, it can not only provide the controller with multi-dimensional network data and adaptively update routing policies, but also reduce the communication and computing pressure on the controller to optimize routing algorithms.

[0018] 2. This invention is based on digital twins and deep reinforcement learning. By separating the training and decision-making processes of deep reinforcement learning, it utilizes historical network environment information from the digital twin to train the routing algorithm and combines an Action-Mask mechanism to reduce the difficulty of exploring the action space, thereby improving the performance of the routing algorithm in complex network environments. This invention can also be extended to other network structures and has a positive effect on building intelligent communication networks. Attached Figure Description

[0019] Figure 1 This is a diagram of the SDN routing optimization technology architecture based on digital twins;

[0020] Figure 2 This is a diagram of the DT-DRL routing algorithm framework;

[0021] Figure 3 This is a flowchart of the Action-Mask mechanism;

[0022] Figure 4 This is a flowchart of the asynchronous DQN algorithm training process. Detailed Implementation

[0023] This invention proposes a digital twin-based SDN routing optimization technique. This technique deeply integrates digital twin technology with deep reinforcement learning algorithms. A deep reinforcement learning model is trained in the digital twin space, and the trained model is used to make routing decisions in the controller. Therefore, by leveraging the efficiency of the digital twin space and the intelligence of deep reinforcement learning, the routing algorithm's intelligent and agile adaptability to the network environment is improved. This is simply referred to as the Digital-Twin and Deep Reinforcement Learning (DT-DRL) based routing algorithm. The technical solution of this invention is further described below with reference to the accompanying drawings.

[0024] like Figure 1As shown, this invention discloses an SDN routing optimization method based on digital twins, comprising two spaces: physical space and virtual space. The data plane, composed of SDN hosts and switches, and the control plane, composed of controllers, exist in the physical space. The digital twin, constructed using host, switch, and link information from the data plane, exists in the virtual space. Network devices such as hosts and switches map their operational status information to the digital twin in real time, supporting the digital twin in achieving accurate network modeling. In the physical space, the control plane is responsible for network traffic forwarding control. The controller provides some computing and communication resources to construct an agent that executes the DT-DRL routing algorithm. This agent is denoted as DRL-Agent. Based on the routing algorithm calculation model and network status information provided by the digital twin, the DRL-Agent plans a suitable forwarding path for each network traffic item. Simultaneously, after traffic transmission is completed, the digital twin collects information such as the actual packet loss rate and latency, feeding this data back into the routing algorithm's training process to further optimize the algorithm. In virtual space, digital twins, based on information provided by virtual twins of network devices, can obtain more accurate and comprehensive network information. This supports more accurate and efficient modeling of the network environment, improves the adaptability of routing algorithms to changes in the network environment, and provides the controller with comprehensive network status data and efficient routing decision algorithms. It also reduces the computational and communication overhead for the controller in sensing network status, analyzing network data, and optimizing routing algorithms. Digital twins can realize functions such as network data collection, digital twin management, and network environment modeling, providing the control plane with services such as network status awareness, network data analysis, and routing model evolution.

[0025] The DT-DRL routing algorithm includes a Policy Network deployed in the controller, a Training Network deployed in the digital twin, and a Target Network for auxiliary training. The digital twin analyzes network status and collects decision-making experience based on historical network data mapped from the data plane, and updates the experience replay pool of the routing algorithm based on the decision-making experience. The routing algorithm is dynamically updated by periodically training the Training Network and updating the parameters of the Policy Network to the controller. When the controller receives a routing request, it uses traffic information and information provided by the digital twin as the environmental status, selects the optimal path for the traffic from multiple alternative paths, and calculates the routing table of the relevant switches based on the path to complete the efficient transmission of traffic.

[0026] Suppose the data plane has C switches and N links. The data plane can be simplified to an undirected graph G = {SW, E}, where SW = {s1, s2, ..., s}. C} represents a set of switches, where s is a set of switches. iLet H be the set of connected hosts. i ={h i，1 h i，2 ,...}, i∈[1, C]; E={e 1，2 e 1，3 , ..., e 2，3 , ...} represents a set consisting of N links, e i,j s i and s j There exists a link, i∈[1,C], j∈[1,C], i≠j; it has a communication distance d i，j Link bandwidth b i，j and link utilization u i,j (t) Three attributes, where d i，j and b i，j u is a constant. i,j (t) represents e i,j The time that data passes through the link per unit of time, and this variable changes with time t.

[0027] Source host h i，j to the destination host h i′，j′ At time t, a traffic request flow is initiated. First, this flow is transmitted to the switch connected to the source host. The flow contains three attributes used for routing decisions: source host address, destination host address, and flow size, where i≠i′, j≠j′. Simultaneously, if the flow is too large, the network uses a packet splitting algorithm to divide it into multiple data packets for transmission. For data packets with the same source and destination host addresses, their transmission paths are identical. After the flow is transmitted, SDN removes the path information for this flow from the switch. The controller then needs to select an appropriate transmission path for each flow. When a flow is transmitted from the previous host or switch to the current switch, the switch needs to forward the flow to the port connecting to the next-hop device according to the path information issued by the controller. If the queue of that port is full, the flow will be discarded; if the queue of that port is not full, the flow can reach the next-hop device after queuing and port processing. Therefore, switch s... i upper port i,j queue length i,j (t) will affect the packet loss rate loss(t) and delay(t) of this flow.

[0028] like Figure 2As shown, the data plane continuously maps network environment information such as hosts, switches, and links to the digital twin. Simultaneously, the digital twin transmits calculated and analyzed network status information to the control plane to support the DRL-Agent in making routing decisions. When a switch in the data plane receives a flow, it reports it to the controller. The controller then uses the network status and flow information provided by the digital twin as the status input to the Policy Network. The action decision output by the Policy Network serves as the optimal transmission path for that flow. After calculation using the routing table, it is sent to the relevant switches in the network, thus completing the transmission of that flow.

[0029] Once a FLW (Flow Log) is transmitted to the destination host, the digital twin can determine a reward value for the DRL-Agent based on the actual packet loss rate and latency of that traffic when mapping the host's operational status data to the digital twin next time. The digital twin will then store the network state space information s, effective decision information a, and reward value r for that decision. t The network state space information s′ at the next moment is used as a decision experience and put into the experience replay pool; the packet loss rate loss(t) is represented by the number of packets Num that reach the destination host in this flow. R Number of packets sent by the source host (Num) T The ratio, i.e.

[0030]

[0031] Once the destination host maps communication data to the corresponding virtual twin in the digital twin, the digital twin can obtain the actual packet loss rate and latency of transmitting that flow.

[0032] When the amount of experience in the experience replay pool meets the preset conditions, the Training Network and Target Network in the digital twin will learn from historical network state changes through experience replay, adjust the neuron parameters in the Training Network and Target Network, and synchronize the parameters in the Training Network with the parameters in the Policy Network in the DRL-Agent to achieve asynchronous updates of the routing algorithm, so as to ensure that the routing algorithm can adapt to the dynamic changes in the network environment.

[0033] To plan suitable transmission paths for traffic in complex and ever-changing network environments, the controller selects the optimal path from K candidate paths based on a deep reinforcement learning routing decision algorithm. This invention uses the DQN algorithm as its foundation, combining Dueling DQN, Double DQN, and priority experience replay mechanisms to improve the learning efficiency of the DQN algorithm. The DT-DRL routing algorithm is based on a deep Q-network, and designs the state space, action space, reward function, and learning process of DQN.

[0034] (1) State space: according to Figure 2 As shown, the DRL-Agent in the control plane can obtain the current network environment status by using the network information provided by the digital twin; the state space is represented by the triple State = <Link, Queue, Flow>, which includes link information Link, port queue information Queue of the switch, and flow information Flow.

[0035] a. Link Information: Includes communication distance D, link bandwidth B, and link utilization U, where D, B, and U represent the communication distance (D), link bandwidth (B), and link utilization (U) of all links in the network, respectively. i,j The distance d between i，j Bandwidth b i，j Link utilization u at time t i,j The set consisting of (t);

[0036] b. Queue Information: Includes queue length (Length) and wait time (Wait), where Length and Wait represent the queue length and wait time respectively, calculated using queuing theory. i,j The average queue length at time t i,j (t) and average packet dwell time wait i，j The set consisting of (t);

[0037] Based on queuing theory, the queuing situation at each switch port is analyzed to obtain the average queue length. i,j (t) and average packet dwell time wait i,j (t) is used to reflect the level of link congestion over a period of time. The average queue length is... i,j (t) and average packet dwell time wait i，jThe calculation method for (t) is as follows: Assume that each switch port has an M / M / 1 / K queue, indicating that the arrival time and transmission delay of each data packet in the port follow a negative exponential distribution. The port can only send one data packet at a time, and the port's queue buffer can hold at most K data packets; where M represents the arrival and transmission times of the data packets, which can be infinitely large, and K is the queue length of the port; based on the switch s... i Mapped to digital twin information, obtain switch s i upper port i,j Let λ be the average number of data packets arriving in 1 second, and let the data packet transmission delay be... Assume p n This represents the probability of the event that the queue length is n (n∈Z). According to queuing theory, when the network is stationary, the probability of the queue length n occurring is p. n =P{N=n}, n=0,1,2,…, and the balance equation of the queue at any time is as follows:

[0038]

[0039] remember Given the average service strength of the port queue, the probability p of having n packets in the port queue is obtained according to formula (2). n for:

[0040] p n =ρ n p0, n = 0, 1, 2, ..., K (3)

[0041] and

[0042]

[0043] According to formulas (3) and (4), the port is obtained. i,j Average queue length i,j (t) is:

[0044]

[0045] Due to the length limitation of the port queue, the effective arrival rate of the queue is:

[0046] λ e =μ(1-p0) (6)

[0047] Using Little's formula, we can obtain the average dwell time (wait) of data packets in the port queue. i，j (t) is:

[0048]

[0049] c. Flow information: A set consisting of the source switch address src, the destination switch address dst, and the flow size;

[0050] (2) Action Space: The action space is the set of all executable actions of the DRL-Agent, denoted by set A:

[0051] A = {a1, a2, ..., a} Z} (8)

[0052] In formula (8), Z = C × (C-1) × K is the number of all alternative paths provided by the controller, a i This indicates any alternative path provided by the controller; for the source switch s src to destination switch s dst path a j have

[0053] a j ={path src,ast |s src ∈SW,s dst ∈SW,s src ≠s dst} (9)

[0054] i∈[1, Z], j∈[1, K];

[0055] The K-shortest path algorithm is used to obtain the s of each pair of source switches. src to destination switch s dst There are K shortest paths between them. However, the number of paths in the Action space increases exponentially with the number of switches. When the number of switches is large, it increases the difficulty for the DQN algorithm to find valid and correct paths. To solve this problem, this invention uses an Action-Mask (exploration pruning) mechanism to mask invalid alternative paths from the transmission flow, thereby improving the performance of the DQN algorithm. The Action-Mask mechanism is as follows: Figure 3 As shown, the specific steps are as follows:

[0056] Step 1: Initialize a matrix Mask with C×(C-1) rows and K columns;

[0057] Step 2: Starting from the first row, iterate through each row of Mask. If the path corresponding to row number i is a valid path corresponding to flow, set the data in that row to 1; otherwise, set all data to 0 and move to the next row to continue the judgment.

[0058] Step 3: After completing Step 2, flatten the Mask into a vector.

[0059] Step4: Utilize and action space vector After performing a dot product, we obtain the effective action 'a'.

[0060] Based on the ∈-greedy strategy, action a is randomly selected from the action space in the initial decision-making stage, which improves the exploration efficiency of the DT-DRL algorithm. The ∈-greedy strategy is shown in formula (10):

[0061]

[0062] In the above formula, Random(K) means randomly selecting one path from K paths as the action, ∈ is a random number, greedy represents the threshold value for action selection, and Q(s, a; w) is the action weight output by PolicyNetwork;

[0063] The action selection strategy combining Action-Mask and ∈-greedy strategies is shown in Equation (11):

[0064]

[0065] In the above formula, Mask() represents the output of the Action-Mask mechanism;

[0066] (3) The purpose of making path decisions in a communication network is to successfully transmit network traffic from the source device to the destination device, and to minimize data loss and communication delay during transmission, so as to provide users with a better communication service experience. Therefore, in order to reduce the packet loss rate loss(t) and communication delay(t) of network traffic, the reward function designed for the DQN algorithm is shown in formula (12):

[0067]

[0068] Where α and β are the weight hyperparameters of packet loss rate loss(t) and delay(t), respectively, and α + β = 1, 0 < α < 1; R f As a penalty for transmission failure; after a routing request for a network traffic flow arrives at the controller at time t, the DRL-Agent, based on the current input network environment state s, executes the DQN algorithm to select action a. When the flow is completed and the destination host completes data mapping to the digital twin, the digital twin will give a reward to the DRL-Agent based on the actual packet loss rate and communication latency of the flow, and record the network environment state s′ at time t+1; the empirical sample (s, a, r) ​​is used to... t ,s′) are added to the experience pool;

[0069] (4) DQN Learning Process: Since the network environment of SDN changes rapidly and dynamically, traditional routing algorithms struggle to respond promptly to these changes. Therefore, this invention proposes using digital twins to perceive the dynamic changes in the network and proposes an asynchronous DQN training method. This ensures timely adjustment of routing policies based on the network environment while also improving the computational speed of the controller in making routing decisions. Because the DQN action space design incorporates the Action-Mask mechanism, when DQN updates the parameters of the Training Network using the experience replay mechanism, the Q-estimated value of the Target Network's output action is:

[0070] y t =r t +γQ[s′,arg max Q(s′,a;w,α,β)′;w - α - ,β - (13)

[0071] Where s′ represents the network environment state at the next time step, α and β are parameters of the Dueling DQN network structure, and w, α, and β represent the network parameters of the Training Network. - α - and β - The network parameters of the Target Network are represented by Q(s′, a; w, α, β)′=Mask(Q(s′, a; w, α, β)). This represents the Action-Mask operation performed on the Q estimate Q(s′, a; w, α, β) of the Target Network output.

[0072] According to the formula for calculating mean square error

[0073]

[0074] Where m is the number of samples, x i y i f(x) i Let represent the sample value, the true value, and the predicted value, respectively. The loss function of the DQN algorithm is defined as:

[0075] L(w)=E{[y t -Q(s, arg max Q(s′, a; w, α, β)′; w, α, β)] 2} (15)

[0076] After calculating the loss function value according to formula (15), backpropagation is performed according to formula (16) to update the parameters of the Training Network:

[0077]

[0078] in, This indicates the gradient calculation operation.

[0079] The specific training process of the DQN algorithm is as follows: Figure 4 As shown, the key steps for implementing the routing optimization algorithm to adaptively update the routing policy according to the network environment are as follows:

[0080] Step 1: First, calculate the set of K paths Path, and initialize the network environment, experience replay pool and neural network parameters. Initialize the neural network parameters by randomly initializing the parameters w in the Training Network and copying w to the Policy Network and Target Network.

[0081] Step 2: When the routing request of the traffic flow arrives at the controller, the DRL-Agent uses the Policy Network to calculate the effective action a according to formula (10), and formulates the routing table based on a and sends it to the switch to select a suitable path for the traffic.

[0082] Step 3: When the destination host maps the traffic transmission information to the digital twin, the digital twin calculates the reward for action a according to formula (12) and assigns the experience sample (s, a, r) ​​to the digital twin. t ,s′) are added to the experience pool;

[0083] Step 4: When the experience playback condition is met, the digital twin selects a batch of learning samples based on the priority experience playback mechanism, updates the parameters w of the Training Network according to formula (16), and synchronously updates the parameters in the Target Network and Policy Network; if the experience playback condition is not met, it is determined whether the network running time has ended. If so, the operation is terminated directly; otherwise, it returns to step 2.

[0084] In summary, the digital twin-based routing optimization algorithm not only utilizes richer network state information for routing decisions but also adjusts routing strategies in a timely manner based on changes in network traffic, thereby ensuring that SDN efficiently plans the optimal transmission path for each network traffic flow. Furthermore, the asynchronous DQN learning method in DT-DRL reduces the storage, computation, and communication resources required by the controller for updating and evolving the DRL-Agent model, improving the controller's routing decision processing efficiency.

[0085] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A digital twin-based SDN routing optimization method, characterized in that, The network consists of two spaces: physical space and virtual space. The data plane, composed of hosts and switches in the SDN architecture, and the control plane, composed of controllers, exist in the physical space. The digital twins built using the host, switch, and link information in the data plane exist in the virtual space. In the physical space, the control plane is responsible for controlling network traffic forwarding. The controllers build agents that execute the DT-DRL routing algorithm. These agents are called DRL-Agents. The DRL-Agents calculate the appropriate forwarding path for each network traffic based on the DT-DRL routing algorithm calculation model and network state information. The DT-DRL routing algorithm includes a Policy Network deployed in the controller, a Training Network deployed in the digital twin, and a Target Network for assisting training. Digital twins analyze network status and collect decision-making experience based on historical network data mapped from the data plane, and update the experience replay pool of routing algorithms based on the decision-making experience. The routing algorithm is dynamically updated by periodically training the Training Network and updating the parameters of the Policy Network to the controller. When the controller receives a routing request, it uses the traffic information and the information provided by the digital twin as the environment status, selects the optimal path for the traffic from multiple alternative paths, and calculates the routing table of the relevant switches based on the path to complete the efficient transmission of traffic. Suppose the data plane has A switch and The links simplify the data plane into an undirected graph. ,in This refers to a set of switches. The set of connected hosts is denoted as , ; Indicates by A collection of links, express and There is a link between them. It has a communication range Link bandwidth and link utilization Three attributes, among which and It is a constant. express The time that data passes through the link per unit of time; this variable varies with time. change; Source host to the destination host At any moment Initiate a traffic request First, request the traffic. The data is transmitted to the switch connected to the source host. It contains three types of data used for routing decisions: source host address, destination host address, and traffic size. ; When the switch in the data plane receives At that time, the switch will report it to the controller, which will then use the network status information provided by the digital twin to... Information serves as the state input to the Policy Network, and the action decisions output by the Policy Network will be used as the basis for that policy. The optimal transmission path is calculated using the routing table and then distributed to the relevant switches in the network, thus completing the route. The transmission; When a After being transmitted to the destination host, when mapping the host's operational status data to the digital twin next time, the digital twin can determine a reward value for the DRL-Agent based on the actual packet loss rate and latency of that traffic, and also allocate network state space information for that decision. Effective decision-making information Reward Value and the network state space information at the next moment This decision-making experience is added to the experience replay pool; packet loss rate This is represented as the clause Number of packets arriving at the destination host Number of packets sent from the source host The ratio, i.e. （1）； When the amount of experience in the experience replay pool meets the preset conditions, the Training Network and Target Network in the digital twin will learn the historical network state changes through experience replay, adjust the neuron parameters in the Training Network and Target Network, and synchronize the parameters in the Training Network with the parameters in the Policy Network in the DRL-Agent to achieve asynchronous updates of the routing algorithm.

2. The SDN routing optimization method based on digital twins according to claim 1, characterized in that, The DT-DRL routing algorithm is based on a deep Q network, and designs the state space, action space, reward function and learning process of DQN. (1) State space: The DRL-Agent in the control plane can obtain the current network environment state using the network information provided by the digital twin; using triples Represents the state space, which contains link information. Switch port queue information and traffic information ; a. Link Information Including communication distance Link bandwidth and link utilization ,in , and These represent all links in the network. Distance between ,bandwidth and time Link utilization The set that constitutes; b. Queue Information Including queue length and stay delay ,in and These represent the results obtained from queuing theory analysis of all switch ports. At any moment Average queue length and average data packet dwell time The set that constitutes; c. Traffic information : Source switch address Destination switch address and flow rate The set that constitutes; (2) Action Space: The action space is a set of all executable actions of the DRL-Agent, represented by a set. express: （2）； In formula (2), The number of all alternative paths provided to the controller. This indicates any alternative path provided by the controller; for the source switch to destination switch path have （3）； ； Adopting based on The shortest path algorithm obtains the shortest path for each pair of source switches. to destination switch Between Find the shortest path; and use an Action-Mask-based mechanism to mask and transmit data. Invalid alternative paths are as follows: Step 1: Initialize a behavior Listed as matrix ; Step 2: Iterate from the first row For each row, if the row number is... The corresponding path is traffic. If the corresponding valid path is specified, the data in that row will be set to 1. Otherwise, set all values ​​to 0 and move to the next line to continue the evaluation; Step 3: After completing Step 2, Flattened into a vector ; Step4: Utilize and action space vector After performing the dot product, the effective action is obtained. ; based on In the initial decision-making phase, the strategy randomly selects actions from the action space. To improve the exploration efficiency of the DT-DRL algorithm, among which The strategy is shown in formula (4): （4）； In the above formula, Indicates from Randomly select one path from the given paths as the action. It is a random number. This indicates the threshold value for action selection. The action weights output by the Policy Network; Combining Action-Mask and The action selection strategy is shown in formula (5): （5）； In the above formula, This indicates the output of the Action-Mask mechanism; (3) The reward function designed for the DQN algorithm is shown in formula (6): （6）； in, and These are packet loss rate and latency The weight hyperparameters, and ; Penalty for transmission failure; at time A network traffic After the routing request reaches the controller, DRL-Agent determines the routing status based on the current network environment. Execute the DQN algorithm to select actions. When the article Once the transmission is complete and the destination host has mapped the data to the digital twin, the digital twin will then... The actual packet loss rate and communication latency are used to assign rewards to the DRL-Agent, and are recorded. Real-time network environment status ; to use empirical samples Add to the experience pool; (4) DQN learning process: Since the action space design of DQN incorporates the Action-Mask mechanism, when DQN updates the parameters of the Training Network using the experience replay mechanism, the Q-estimated value of the Target Network's output action is: （7）； in, Indicates the network environment status at the next moment. and These are the parameters of the Dueling DQN network structure. , and This represents the network parameters of the Training Network. , and Indicates the network parameters of the Target Network. This represents the Q estimate of the output of the Target Network. Perform an Action-Mask operation; According to the formula for calculating mean square error （8）； in It is the sample size. Let the sample value, the true value, and the predicted value be represented respectively. The loss function of the DQN algorithm is defined as: （9）； After calculating the loss function value according to formula (9), backpropagation is performed according to formula (10) to update the parameters of the TrainingNetwork: （10）； in, This indicates the gradient calculation operation.

3. The SDN routing optimization method based on digital twins according to claim 2, characterized in that, The average queue length and average data packet dwell time The calculation method is as follows: assuming the queue size for each switch port is... A queue indicates that the arrival time and transmission delay of each data packet in the port follow a negative exponential distribution. The port can only send one data packet at a time, and the port's queue buffer can hold at most one packet. 1 data packet; among which This indicates that the arrival and transmission times of data packets can be infinitely large. The queue length for the port; depending on the switch. Information mapped to a digital twin is used to obtain the switch. Up port The average number of data packets arriving in 1 second is The data packet transmission delay is ;assumed Indicates that the queue length is The probability of this event occurring. According to queuing theory, when the network is stable, the queue length is... The probability of occurrence is The balance equation of the queue at any given time is as follows: （11）； remember The average service strength of the port queue is calculated according to formula (11). The probability of a data packet for: （12）； and （13）； According to formulas (12) and (13), the port is obtained. Average queue length for: （14）； Due to the length limitation of the port queue, the effective arrival rate of the queue is: （15）； Using Little's formula, we can obtain the average dwell time of data packets in the port queue. for: （16）。

Images