SDN (Software Defined Network) routing optimization method and device based on deep reinforcement learning

Abstract

The invention discloses an SDN routing optimization method and device based on deep reinforcement learning, and the method comprises the steps: collecting network state data in a data center network, carrying out the data preprocessing of the network state data, and enabling the network state data to comprise the IP addresses of a source host and a destination host, the size of a data flow, the type of the data flow, and the weight of the data flow; training the target algorithm model, and optimizing and adjusting a hyper-parameter combination through a Bayesian optimization technology to form a routing optimization model; performing routing calculation by using the routing optimization model to obtain a routing calculation result; and a routing optimization decision is made based on a routing calculation result, so that the problems of low network utilization rate, congestion, time delay and the like caused by long convergence time and high resource occupancy rate of a traditional routing optimization algorithm limited by an algorithm basis and a working principle are solved, and the routing efficiency and the network utilization rate in the data center network are improved.

Description

Technical Field

[0001] The present application relates to the field of network management technology, and more specifically, to a SDN routing optimization method and device based on deep reinforcement learning. Background Art

[0002] With the rapid growth of the digital economy, data centers have become important infrastructure in the fields of artificial intelligence, big data, cloud computing, and the Internet. The research on the internal network of the data center (DCN) has now become a hot spot for technological innovation and application. Traditional network structures are difficult to flexibly and in real time configure resources in large-scale, high-density traffic data center networks. Software Defined Network (SDN) technology decouples the control plane and the data forwarding plane, enhancing the flexibility of the network structure. In the DCN under the SDN architecture, the routing algorithm is required to obtain the status information of the global network and formulate the forwarding path planning of high-density traffic in the dynamically changing complex network topology, and to be able to make real-time and flexible adjustments for emergencies. The traditional routing optimization algorithm is limited by its algorithm foundation and working principle, and has problems such as long convergence time and high resource occupancy, resulting in low network utilization, congestion and delay, and it is difficult to meet the above requirements. Therefore, innovative research is urgently needed to solve the routing optimization problem of DCN under the SDN architecture under the new situation.

[0003] Deep reinforcement learning (DRL), as an emerging machine learning technology, provides new ideas for SDN routing optimization by combining the representation learning ability of deep learning and the decision-making ability of reinforcement learning. DRL can continuously interact with the environment to learn, automatically optimize routing strategies, adapt to the dynamic changes of network traffic, and show higher flexibility and adaptability.

[0004] However, although the relevant routing optimization methods have theoretical advantages, they face challenges in the application scenarios of data center networks: first, the DDPG algorithm is extremely sensitive to the selection of hyperparameters, and inappropriate parameter settings may significantly reduce the performance of the algorithm; second, the method may face the problems of slow convergence and insufficient stability, affecting the optimization efficiency and the reliability of the results; third, the design and implementation of high-precision network measurements and new source routing protocols have high requirements on network equipment and technologies, which increases the difficulty of application; finally, the dynamic setting of link weights may lag when the network status changes rapidly, requiring further algorithm improvement and adaptive adjustment. Summary of the invention

[0005] In response to at least one defect or improvement need in the prior art, the present invention provides a SDN routing optimization method and device based on deep reinforcement learning, which solves the problems of long convergence time and high resource occupancy rate of traditional routing optimization algorithms due to the limitations of their algorithm basis and working principles, resulting in low network utilization, congestion and delay, etc., and realizes the improvement of routing efficiency and network utilization in data center networks.

[0006] To achieve the above-mentioned purpose, according to a first aspect of the present invention, a SDN routing optimization method based on deep reinforcement learning is provided, the method comprising: collecting network status data in a data center network, and performing data preprocessing on the network status data, wherein the network status data comprises IP addresses of a source host and a destination host, data flow size, data flow type, and data flow weight; training a target algorithm model, optimizing and adjusting a hyperparameter combination through Bayesian optimization technology, and forming a routing optimization model; performing routing calculation using the routing optimization model to obtain routing calculation results; and making routing optimization decisions based on the routing calculation results.

[0007] In an exemplary embodiment, the collection of network status data in a data center network and data preprocessing include: applying an exponential function to each element in an original array to map any real number to a positive real number range to obtain a target array; calculating the sum of all elements in the target array, dividing each element in the original array by the sum of all element exponentials to obtain a softmax value for each element; and converting the original array into a target probability distribution, wherein the probability of each element is proportional to its relative size in the original array.

[0008] In an exemplary embodiment, the target algorithm model is trained and the hyperparameter combination is optimized and adjusted through Bayesian optimization technology to form a routing optimization model, including: adopting experience replay technology and using TD error value to measure the learning value of each historical data; selecting historical data with high reward value or high penalty value to train and update the target algorithm model.

[0009] In an exemplary embodiment, the use of the routing optimization model to perform routing calculations to obtain routing calculation results includes: processing a state sequence in an LSTM layer before the first linear layer of an Actor network and a Critic network, generating an implicit state and outputting a vector of a fixed dimension; and using the vector as input to a subsequent linear layer and inputting it into a neural network in an Actor and Critic architecture, wherein the Actor network is used to determine an action strategy under each state.

[0010] In an exemplary embodiment, after the application receives the signal and registers an interrupt service function after the thread that generates the random number, the method further includes: initializing the weights of the Critic network by uniform distribution, and the hidden layer uses a rectified linear unit activation function; the output layer of the Actor network uses a hyperbolic tangent function to limit the output range of the action and maps the action value into a target range.

[0011] In an exemplary embodiment, making a routing optimization decision based on the routing calculation result includes: using an objective function to create a generator, the generator generates all shortest simple paths from a source node to a destination node, wherein the simple path is a path that does not contain duplicate nodes; storing the shortest simple path and combining it with the target network state to obtain the routing optimization decision.

[0012] According to the second aspect of the present invention, there is also provided an SDN routing optimization device based on deep reinforcement learning, which includes: a preprocessing unit, used to collect network status data in a data center network, and perform data preprocessing on the network status data, wherein the network status data includes the IP addresses of the source host and the destination host, the data flow size, the data flow type, and the data flow weight; a training unit, used to train the target algorithm model, and optimize and adjust the hyperparameter combination through Bayesian optimization technology to form a routing optimization model; a calculation unit, used to perform routing calculation using the routing optimization model to obtain a routing calculation result; an optimization decision unit, used to make a routing optimization decision based on the routing calculation result.

[0013] According to a third aspect of the present invention, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned SDN routing optimization method based on deep reinforcement learning when running.

[0014] According to a fourth aspect of the present invention, there is also provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the SDN routing optimization method based on deep reinforcement learning through the computer program.

[0015] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0016] (1) The present invention provides an SDN routing optimization method based on deep reinforcement learning. In the data input stage, the original data of the network status is preprocessed by the softmax function to unify the data format and facilitate feature extraction, which can effectively improve the speed of the algorithm model in processing data and accelerate the entire routing optimization process. In the model training stage, by introducing Bayesian optimization technology and using Gaussian process as a priori model, the hyperparameter combination consisting of the learning rate, discount factor, and experience replay pool size in the model is optimized to solve the problem that the DDPG basic model is sensitive to hyperparameter adjustment.

[0017] (2) By adopting a SDN routing optimization method based on deep reinforcement learning provided by the present invention, in the data processing stage, by designing a priority experience replay pool, based on the temporal difference error (TD error), different sampling probabilities representing importance are assigned to samples, so that the system can pay more attention to those experience samples that have a significant impact on performance improvement, thereby helping the system to converge to a better routing strategy more quickly. This learning mechanism helps the model to use historical data more effectively and improve learning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of a flow chart of an optional SDN routing optimization method based on deep reinforcement learning provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of another optional SDN routing optimization method based on deep reinforcement learning provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of the structure of an optional SDN routing optimization device based on deep reinforcement learning provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0024] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0025] According to one aspect of the embodiments of the present application, a SDN routing optimization method based on deep reinforcement learning is provided. Figure 1 The present invention describes an SDN routing optimization method based on deep reinforcement learning.

[0026] Figure 1 is a flow chart of an optional SDN routing optimization method based on deep reinforcement learning provided in an embodiment of the present application, such as Figure 1 As shown, the process of the method may include the following steps:

[0027] S102, collecting network status data in the data center network and performing data preprocessing on the network status data, wherein the network status data includes IP addresses of source hosts and destination hosts, data flow size, data flow type, and data flow weight;

[0028] S104, training the target algorithm model, optimizing and adjusting the hyperparameter combination through Bayesian optimization technology, and forming a routing optimization model;

[0029] S106, performing route calculation using the route optimization model to obtain a route calculation result;

[0030] S108: Make a route optimization decision based on the route calculation result.

[0031] The embodiment of the present application provides an SDN routing optimization method based on deep reinforcement learning. Starting from the characteristics and requirements of the data center network application scenario, it aims to solve the difficult problems of low flexibility and poor reliability of traditional routing optimization algorithms under high-density traffic and dynamic topology changes.

[0032] Figure 2 A flowchart of another optional SDN routing optimization method based on deep reinforcement learning provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the technical solution adopted by the present invention can be divided into three layers according to the function in design, namely, data input layer, data processing and routing selection layer, and routing decision layer. The specific process is to first set the input and preprocess the data; secondly, set the priority experience replay pool; then add the LSTM neural network layer, and adjust the replay pool data input form; then implement the Bayesian optimization adjustment hyperparameter module, and finally obtain the function realization.

[0033] Optionally, a "D+L" type routing optimization method based on a deep deterministic policy gradient (DDPG) algorithm model and a long short-term memory (LSTM) neural network structure in the context of a data center network can be adopted, and implemented through the following steps: collecting network status data in the data center network and performing data preprocessing; training the "D+L" algorithm model (i.e., the target model), and optimizing and adjusting the hyperparameter combination through Bayesian optimization technology to form a mature routing optimization model; performing routing calculations using the trained "D+L" algorithm model; and making routing optimization decisions based on the routing calculation results.

[0034] Through the above steps S102 to S108, the network status data in the data center network is collected and data preprocessed, wherein the network status data includes the IP addresses of the source host and the destination host, the size of the data flow, the type of the data flow, and the weight of the data flow; the target algorithm model is trained, and the hyperparameter combination is optimized and adjusted by the Bayesian optimization technology to form a routing optimization model; the routing optimization model is used to perform routing calculations to obtain routing calculation results; and routing optimization decisions are made based on the routing calculation results, thereby solving the problems of long convergence time and high resource occupancy rate of traditional routing optimization algorithms due to the limitations of their algorithm basis and working principles, resulting in low network utilization, congestion and delay, etc., and achieving the improvement of routing efficiency and network utilization in the data center network.

[0035] In an exemplary embodiment, collecting network status data in a data center network and performing data preprocessing includes:

[0036] S11, applying the exponential function to each element in the original array, mapping any real number to the positive real number range to obtain the target array;

[0037] S12, calculate the sum of all elements in the target array, divide each element in the original array by the sum of all element indices, and obtain the softmax value of each element;

[0038] S13, converting the original array into a target probability distribution, wherein the probability of each element is proportional to its relative size in the original array.

[0039] In this embodiment, during the network status data collection and preprocessing process, the network status data includes the IP addresses of the source host and the destination host, the data flow size, the data flow type, and the data flow weight. During data preprocessing, an exponential function (e to the power of x) can be first applied to each element in the array X to map any real number to the positive real number range; then the sum of all elements in the array obtained in the previous step is calculated; finally, each element in the original array is divided by the sum of all element exponentials to obtain the softmax value of each element. The original array is converted into a probability distribution, where the probability of each element is proportional to its relative size in the original array.

[0040] Using the softmax function for data preprocessing can improve data utilization. The softmax function maps different raw scores to the same scale through exponentialization and normalization, which helps to deal with the scale differences that may exist in the raw scores of different categories. By converting the raw scores into probability distributions, the softmax function provides an intuitive and effective way to deal with multi-classification problems. Due to the differentiability of the softmax function, the process of calculating the gradient of the loss function with respect to the model output will become more convenient. Compared with directly applying the algorithm model without data preprocessing, it is more convenient and efficient in extracting data features and feature patterns, which can improve the utilization efficiency of data and the calculation speed of the model.

[0041] Through this embodiment, in the data input stage, the original data of the network status is preprocessed by the softmax function, the data format is unified, and feature extraction is facilitated, which can effectively improve the speed of the algorithm model in processing data and accelerate the entire routing optimization process.

[0042] In an exemplary embodiment, training the target algorithm model and optimizing and adjusting the hyperparameter combination by Bayesian optimization technology to form a routing optimization model includes:

[0043] S21, adopt the experience replay technology and use the TD error value to measure the learning value of each historical data; S22, select historical data with high reward value or high penalty value to train and update the target algorithm model.

[0044] In this embodiment, during the training of the "D+L" algorithm model, the SDN network includes m links between n switches, generating d types of data flows, and the state is the traffic matrix of different types of data flows between network node pairs, as well as the link utilization matrix and delay matrix between network node pairs, as shown in the following formula:

[0045] s=[T,U,D],

[0046] Where T=[T1,T2,...,T d ] represents the traffic matrix of different types of data flows in the network, describing the transmission requirements of a certain type of data flow in the network. i =(i=1,2,...,d) is shown in the following formula:

[0047]

[0048] Among them, f ij is the amount of traffic transmitted from switch i to switch j.

[0049] U represents the link utilization matrix between node pairs in the network, describing the link usage between switches, as shown in the following formula:

[0050]

[0051] Among them, u ij is the link utilization between switch i and switch j. If there is no link connection between switches, then u ij =-1.

[0052] D represents the delay matrix between node pairs in the network, describing the link delay between switches, as shown in the following formula:

[0053]

[0054] d ij is the link delay between switch i and switch j. If there is no link connection between switches, then d ij =-1.

[0055] The action space design is as follows:

[0056] a=[w1,w2,...,w m ],

[0057] Among them, w i Represents the weight value of the i-th link when making routing decisions.

[0058] The reward is a Qos evaluation of the benefit obtained from the last action performed by the agent. The routing optimization goal is to minimize the transmission cost of the path. Therefore, the transmission cost of the path is used as the reward function, as shown in the following formula:

[0059] r=-cost

[0060] During the model training process, the experience replay technology is adopted to use the TD error value to measure the learning value of each historical data, and historical data with high reward values ​​or high penalty values ​​are preferentially selected to train and update the neural network. The TD error value δ of the jth historical data j The calculation of is as follows:

[0061] δ j = r(s t ,a t )+γQ'(s t+1 ,a t+1 ,w)-Q(s t ,a t ,w)

[0062] Based on the Actor-Critic structure, it relies on deep neural networks to simulate the policy network and approximate the action value function. The soft update strategy is adopted to update the parameters in the two target networks. Random noise is added to the output of the action network to explore a wider action space. The stochastic gradient method is used to optimize the relevant parameters during the training process. The entire training process is used as a sample, and the Gaussian process is used as the prior model. The hyperparameter combination consisting of the learning rate, discount factor, and experience replay pool size in the input model and the training performance of the combination in the model are calculated together with the input and the prior model to obtain the posterior distribution. The update is performed based on the calculation results of the posterior distribution and the input, and then the update result is applied to the algorithm model.

[0063] Through the design of the priority experience replay pool, different sampling probabilities representing importance are assigned to samples based on the temporal difference error (TD error), so that the system can pay more attention to those experience samples that have a significant impact on performance improvement, thereby helping the system to converge to a better routing strategy more quickly. This learning mechanism helps the model to use historical data more effectively and improve learning efficiency.

[0064] Through this embodiment, a priority experience replay pool is designed to improve sample utilization. The TD error value is used to measure the learning value of each historical data, and the priority is set. During training, samples that are more critical to improving network performance are given priority, which helps the model to use historical data more effectively, improve learning efficiency, reduce training fluctuations, and accelerate the convergence process. Compared with an experience replay pool without priority, this design can effectively improve the utilization efficiency of samples.

[0065] In an exemplary embodiment, the route calculation is performed using the route optimization model to obtain the route calculation result including:

[0066] S31, the LSTM layer before the first linear layer of the Actor network and the Critic network processes the state sequence, generates a hidden state and outputs a vector of fixed dimension;

[0067] S32, the vector is input into the neural network in the Actor and Critic architecture as the input of the subsequent linear layer, where the Actor network is used to determine the action strategy under each state.

[0068] In this embodiment, in the process of performing routing calculation using the trained "D+L" algorithm model, optionally, the state sequence can be processed in the LSTM layer before the first linear layer of the Actor and Critic network to generate a hidden state and output a vector of fixed dimension as the input of the subsequent linear layer and input to the neural network in the Actor and Critic architecture.

[0069] The Actor network is used to determine the action strategy for a given state. Specifically, the first layer of the network, the L1 linear layer, maps the input state dimension from state_dim to 256 dimensions. The second layer, the L2 linear layer, keeps the output dimension of the previous layer at 256 dimensions. The third layer, L3, reduces the dimension from 256 dimensions to 64 dimensions. The L4MF linear layer maps the output of L3 to 10 dimensions, and each linear layer maps the output of L4MF to the kPath dimension. There is a loop in the code that creates these layers based on the result of dividing path_dim (path dimension) by 3. The weights of the network are initialized using a uniform distribution.

[0070] The Critic network is used to evaluate the value function of the state-action pair under the current strategy. The first layer of the network, L1, is a linear layer that maps the input state dimension from state_dim to 256 dimensions. The second layer, L2, maps the output dimension of the previous layer to the same dimension as the action dimension. The third layer, L3, concatenates the two inputs (state and action) and reduces the dimension from 2 times the action dimension to 8 dimensions through a linear layer. The fourth layer, L4, maps the output of L3 to 1 dimension, that is, obtains the value estimate of the state-action pair.

[0071] Through this embodiment, in the data processing stage, in the Actor network and Critic network of the DDPG basic model, an LSTM layer is added before the first linear layer to process the state sequence and capture the temporal dependencies in the network state, thereby helping the system to make more accurate and effective routing choices. The LSTM layer is added to improve the processing capabilities of time-related data. The LSTM neural network structure can remember and utilize past information, effectively capture the temporal dependencies in the network state, and enhance the model's ability to process time series data and remember long-term dependency information. Taking into account previous network state information when performing route optimization helps to improve the robustness and reliability of route optimization decisions.

[0072] In an exemplary embodiment, after the application receives a signal, registering an interrupt service function after the thread generating the random number, the method further includes:

[0073] S41, the critic network weights are initialized by uniform distribution, and the hidden layer uses the rectified linear unit activation function;

[0074] S42, the output layer of the Actor network uses the hyperbolic tangent function to limit the output range of the action and map the action value into the target range.

[0075] In this embodiment, the weights of the Critic network are initialized by uniform distribution. All hidden layers (except the output layer) use the rectified linear unit (ReLU) activation function. The output layer of the Actor network uses the hyperbolic tangent function (tanh) to limit the output range of the action and map the action value to the range of [-1,1].

[0076] Through this embodiment, in the data processing stage, in the Actor network and the Critic network of the DDPG basic model, an LSTM layer is added before the first linear layer to process the state sequence, so as to capture the temporal dependency in the network state, thereby helping the system to make more accurate and effective routing selection.

[0077] In an exemplary embodiment, making a route optimization decision based on the route calculation result includes:

[0078] S51, using the target function to create a generator, the generator will generate all the shortest simple paths from the source node to the destination node, wherein the simple path is a path that does not contain repeated nodes;

[0079] S52, the shortest simple path is stored, and combined with the target network state, a routing optimization decision is obtained.

[0080] In this embodiment, the way to make a route optimization decision from the route calculation result can be to use the shortest_simple_paths function of NetworkX to create a generator, which generates all the shortest simple paths from the source node src to the destination node dst. A simple path refers to a path that does not contain repeated nodes. These paths are then stored in paths[src][dst], and the final route optimization decision is made in combination with the actual network status.

[0081] Through this embodiment, Bayesian optimization technology is introduced to adjust hyperparameters and reduce the sensitivity of the algorithm. Using Gaussian process as the prior model, the hyperparameter combination consisting of the learning rate, discount factor, and experience replay pool size in the DDPG model and the training performance of the combination in the DDPG model are input, and the input is calculated together with the prior model to obtain the posterior distribution, and the hyperparameter combination data is updated according to the calculation results of the posterior distribution and the input. The sensitivity of the DDPG algorithm model to hyperparameter adjustment can be alleviated to a certain extent, thereby improving the overall robustness of the "D+L" routing optimization algorithm, and avoiding violent shocks during routing optimization that lead to poor convergence.

[0082] According to another aspect of an embodiment of the present application, an SDN routing optimization device for implementing the above-mentioned SDN routing optimization method based on deep reinforcement learning is also provided. Figure 3 is a schematic diagram of the structure of an optional SDN routing optimization device based on deep reinforcement learning according to an embodiment of the present application, such as Figure 3 As shown, the device may include:

[0083] A preprocessing unit 302 is used to collect network status data in the data center network and perform data preprocessing on the network status data, wherein the network status data includes the IP addresses of the source host and the destination host, the data flow size, the data flow type, and the data flow weight;

[0084] The training unit 304 is used to train the target algorithm model, optimize and adjust the hyperparameter combination through the Bayesian optimization technology, and form a routing optimization model;

[0085] A calculation unit 306 is used to perform route calculation using the route optimization model to obtain a route calculation result;

[0086] The optimization decision unit 308 is used to make a route optimization decision based on the route calculation result.

[0087] It should be noted that the preprocessing unit 302 in this embodiment can be used to execute the above step S102, the training unit 304 in this embodiment can be used to execute the above step S104, the calculation unit 306 in this embodiment can be used to execute the above step S106, and the optimization decision unit 308 in this embodiment can be used to execute the above step S108.

[0088] Through the above modules, the network status data in the data center network is collected and preprocessed, wherein the network status data includes the IP addresses of the source host and the destination host, the size of the data flow, the type of the data flow, and the weight of the data flow; the target algorithm model is trained, and the hyperparameter combination is optimized and adjusted through the Bayesian optimization technology to form a routing optimization model; the routing optimization model is used to perform routing calculations to obtain routing calculation results; routing optimization decisions are made based on the routing calculation results, which solves the problems of long convergence time and high resource occupancy rate of traditional routing optimization algorithms due to the limitations of their algorithm foundation and working principles, resulting in low network utilization, congestion and delay, etc., and realizes the improvement of routing efficiency and network utilization in the data center network.

[0089] In an exemplary embodiment, the pre-processing unit comprises:

[0090] A mapping module is used to apply an exponential function to each element in the original array, mapping any real number to the positive real number range to obtain a target array;

[0091] A calculation module, used for calculating the sum of all elements in the target array, dividing each element in the original array by the sum of all element exponents, and obtaining a softmax value of each element;

[0092] A conversion module is used to convert the original array into a target probability distribution, wherein the probability of each element is proportional to its relative size in the original array.

[0093] In an exemplary embodiment, the training unit comprises:

[0094] The learning module is used to adopt the experience replay technology and use the TD error value to measure the learning value of each historical data;

[0095] The updating module is used to select historical data with high reward values ​​or high penalty values ​​to train and update the target algorithm model.

[0096] In an exemplary embodiment, the computing unit comprises:

[0097] The generation module is used to process the state sequence in the LSTM layer before the first linear layer of the Actor network and the Critic network, generate hidden states and output a vector of fixed dimension;

[0098] An input module is used to input the vector as an input of a subsequent linear layer into a neural network in an Actor and Critic architecture, wherein the Actor network is used to determine an action strategy in each state.

[0099] In an exemplary embodiment, the apparatus further comprises:

[0100] Initialization unit, used to initialize the critic network weights through uniform distribution, and the hidden layer uses the rectified linear unit activation function;

[0101] The mapping unit, used in the output layer of the Actor network, uses the hyperbolic tangent function to limit the output range of the action and map the action value to the target range.

[0102] In an exemplary embodiment, the optimization decision unit includes:

[0103] A creation module is used to create a generator using an objective function, and the generator generates all shortest simple paths from a source node to a destination node, wherein the simple path is a path that does not contain repeated nodes; an acquisition module is used to store the shortest simple path and obtain the routing optimization decision in combination with the target network state.

[0104] It should be noted here that the examples and scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiments. It should be noted that the above-mentioned modules as part of the device can run in a hardware environment and can be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0105] According to another aspect of the embodiments of the present application, a storage medium is further provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the above-mentioned SDN routing optimization methods based on deep reinforcement learning in the embodiments of the present application.

[0106] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps:

[0107] S1, collecting network status data in the data center network and performing data preprocessing on the network status data, wherein the network status data includes the IP addresses of the source host and the destination host, the data flow size, the data flow type, and the data flow weight;

[0108] S2, training the target algorithm model, optimizing and adjusting the hyperparameter combination through Bayesian optimization technology, and forming a routing optimization model;

[0109] S3, using the routing optimization model to perform routing calculation and obtain routing calculation results;

[0110] S4, making a route optimization decision based on the route calculation result.

[0111] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0112] Among them, computer-readable storage media may include, but are not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0113] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned SDN routing optimization method based on deep reinforcement learning is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0114] Figure 4 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Figure 4 As shown, it includes a processor 402, a communication interface 404, a memory 406 and a communication bus 408, wherein the processor 402, the communication interface 404, and the memory 406 communicate with each other through the communication bus 408, wherein:

[0115] Memory 406, used to store computer programs;

[0116] The processor 402 is used to implement the following steps when executing the computer program stored in the memory 406:

[0117] S1, collecting network status data in the data center network and performing data preprocessing on the network status data, wherein the network status data includes the IP addresses of the source host and the destination host, the data flow size, the data flow type, and the data flow weight;

[0118] S2, training the target algorithm model, optimizing and adjusting the hyperparameter combination through Bayesian optimization technology, and forming a routing optimization model;

[0119] S3, using the routing optimization model to perform routing calculation and obtain routing calculation results;

[0120] S4, making a route optimization decision based on the route calculation result.

[0121] Optionally, the communication bus may be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4The communication interface is used for communication between the electronic device and other devices.

[0122] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0123] As an example, the memory 406 may include, but is not limited to, the preprocessing unit 302, the training unit 304, the calculation unit 306, and the optimization decision unit 308 in the SDN route optimization device based on deep reinforcement learning. In addition, it may also include, but is not limited to, other module units in the SDN route optimization device based on deep reinforcement learning, which will not be repeated in this example.

[0124] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0125] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0126] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0127] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0128] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0129] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0131] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program code.

[0132] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0133] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0134] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A SDN routing optimization method based on deep reinforcement learning, characterized in that: include: Collecting network status data in the data center network and performing data preprocessing on the network status data, wherein the network status data includes the IP addresses of the source host and the destination host, the data flow size, the data flow type, and the data flow weight; Train the target algorithm model, optimize and adjust the hyperparameter combination through Bayesian optimization technology, and form a routing optimization model; Using the routing optimization model to perform routing calculation to obtain routing calculation results; A route optimization decision is made based on the route calculation result.

2. The SDN routing optimization method based on deep reinforcement learning according to claim 1, characterized in that: The collecting of network status data in the data center network and performing data preprocessing includes: Apply the exponential function to each element in the original array to map any real number to the positive real number range to obtain the target array; Calculate the sum of all elements in the target array, divide each element in the original array by the sum of all element exponents, and obtain the softmax value of each element; The original array is converted to a target probability distribution, where the probability of each element is proportional to its relative size in the original array.

3. The SDN routing optimization method based on deep reinforcement learning according to claim 1, characterized in that: The training of the target algorithm model and the optimization and adjustment of the hyperparameter combination by Bayesian optimization technology to form a routing optimization model include: Adopt experience replay technology and use TD error value to measure the learning value of each historical data; Select historical data with high reward values ​​or high penalty values ​​to train and update the target algorithm model.

4. The SDN routing optimization method based on deep reinforcement learning according to claim 1, characterized in that: The method of performing route calculation using the route optimization model to obtain a route calculation result includes: The LSTM layer before the first linear layer of the Actor network and the Critic network processes the state sequence, generates a hidden state and outputs a vector of fixed dimension; The vector is used as the input of the subsequent linear layer and input into the neural network in the Actor and Critic architecture, where the Actor network is used to determine the action strategy under each state.

5. The SDN routing optimization method based on deep reinforcement learning according to claim 4, characterized in that: After the application receives the signal, registering an interrupt service function after the thread generating the random number, the method further comprises: The critic network weights are initialized by uniform distribution, and the hidden layer uses the rectified linear unit activation function; The output layer of the Actor network uses a hyperbolic tangent function to limit the output range of the action and map the action value into the target range.

6. The SDN routing optimization method based on deep reinforcement learning according to claim 1, characterized in that: The making of a route optimization decision based on the route calculation result comprises: Using the objective function, a generator is created, wherein the generator generates all shortest simple paths from a source node to a destination node, wherein the simple path is a path that does not contain repeated nodes; The shortest simple path is stored and combined with the target network state to obtain the routing optimization decision.

7. A SDN routing optimization device based on deep reinforcement learning, characterized in that: include: A preprocessing unit, used to collect network status data in the data center network and perform data preprocessing on the network status data, wherein the network status data includes the IP addresses of the source host and the destination host, the data flow size, the data flow type, and the data flow weight; The training unit is used to train the target algorithm model and optimize and adjust the hyperparameter combination through Bayesian optimization technology to form a routing optimization model; A calculation unit, used to perform routing calculation using the routing optimization model to obtain routing calculation results; The optimization decision unit is used to make a route optimization decision based on the route calculation result.

8. The SDN routing optimization device based on deep reinforcement learning according to claim 7, characterized in that: The pre-processing unit comprises: A mapping module is used to apply an exponential function to each element in the original array, mapping any real number to the positive real number range to obtain a target array; A calculation module, used for calculating the sum of all elements in the target array, dividing each element in the original array by the sum of all element exponents, and obtaining a softmax value of each element; A conversion module is used to convert the original array into a target probability distribution, wherein the probability of each element is proportional to its relative size in the original array.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 6 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

Images