A dqn-based optical network routing and spectrum allocation algorithm

Abstract

The application discloses a kind of DQN-based optical network routing and spectrum allocation algorithm, comprising steps S1: defining optical network environment and algorithm initialization, the basic framework of DQN model is constructed;Step S2: receiving optical path request, executing RSA decision;Step S3: according to the reward of the RSA decision feedback executed, and the DQN model is trained;Step S4: the DQN model is tested, according to the current network state for each received optical path request, the optimal path and spectrum allocation scheme are selected.This application discloses a kind of DQN-based optical network routing and spectrum allocation algorithm, in the frequency of training and target network update in spectrum allocation, algorithm, and the update of exploration rate etc. are optimized and improved, can improve the problem of higher blocking rate, lower spectrum resource utilization rate etc. existing in traditional heuristic algorithm.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of optical network resource configuration, and particularly relates to an optical network routing and spectrum allocation (RSA) algorithm based on a deep Q-network (DQN). BACKGROUND

[0002] With the rapid development of information technology, optical networks as a new type of optical network technology have become an indispensable part of modern society. However, with the expansion of network size and the increase of user demand, how to efficiently perform routing and spectrum allocation has become a key problem to be solved in the field of optical networks.

[0003] In the paper "Online routing and spectrum allocation in elastic optical networks based on dueling Deep Q-network", RSA is proposed to establish an optical path for each traffic demand by selecting a suitable route and allocating sufficient spectrum, that is, to establish an optical path connection between the source node and the destination node and allocate an appropriate number of contiguous, continuous, and non-overlapping frequency slots (FSs). Among them, contiguous refers to a contiguous constraint, which requires that the frequency band allocated to a traffic demand must consist of contiguous FSs; continuous refers to a continuity constraint, which requires that on each link of a traffic demand path, contiguous FSs with the same position index must be allocated; and non-overlapping indicates a non-overlapping constraint, that is, an occupied FS cannot be allocated to another traffic demand.

[0004] In optical networks, the making of RSA decisions relies on a comprehensive consideration of network status, traffic demand, and available spectrum resources, which makes it difficult to make RSA decisions. In addition, because the solution space of the RSA problem is huge, traditional exhaustive methods are not powerful enough to find the optimal solution in a reasonable time when facing large-scale networks; traditional heuristic algorithms such as the Greedy algorithm and the KPaths algorithm introduced in the paper “Study and Analysis of Routing and Spectrum Allocation (RSA) and Routing, Modulation and Spectrum Allocation (RMSA) Algorithms in Elastic Optical Networks (EONs)” are prone to local optimal solutions and cannot find global optimal solutions, which can lead to low spectrum utilization and other problems, thereby affecting the overall performance of the network; and traditional methods are often difficult to adapt to complex and changing network environments. Therefore, a method is needed that can intelligently adapt to network changes and efficiently solve the RSA problem.

[0005] The paper “Deep reinforcement learning for comprehensive route optimization in elastic optical networks using generative strategies” uses reinforcement learning algorithm and combines with generative strategy (GS) to reduce the search range of action space and improve the efficiency and accuracy of decision-making, but at the same time, the introduction of generative strategy also increases the computational complexity of the model, and the GS model cannot adapt to more complex problems as the author previously expected. The paper “Online routing and spectrum allocation in elastic optical networks based on dueling deep q-network” uses the Dueling network architecture in reinforcement learning method to decompose the Q value into state value and action advantage, which helps the network to learn and take action faster and improve the learning efficiency, but the training complexity of the model will also increase accordingly.

[0006] Therefore, in view of the above problems, further improvements are needed. SUMMARY

[0007] The main purpose of the present application is to provide a DQN-based optical network routing and spectrum allocation algorithm, which optimizes and improves the frequency of training and target network update, the update of exploration rate and other aspects in spectrum allocation algorithm, and can improve the problems of high blocking rate and low spectrum resource utilization rate existing in traditional heuristic algorithms (such as Greedy algorithm, KPaths algorithm).

[0008] Another purpose of the present application is to provide a DQN-based optical network routing and spectrum allocation algorithm, which selects to combine the reinforcement learning algorithm with the RSA problem in the elastic optical network, and can be an effective method for optimizing the RSA problem. Therefore, the present application adopts the DQN algorithm in the reinforcement learning method, and makes other optimizations in other aspects such as considering the influence of spectrum fragmentation on the basis of the traditional algorithm.

[0009] The traditional DQN algorithm is to extract features and approximate value functions of complex network states through a deep neural network. The present application defines the preferred state space, such as the source node-destination node, bandwidth information contained in the optical path request, link and spectrum resource information in the optical network environment, etc.; the action space, such as selecting a certain path, allocating a certain spectrum, etc.; the reward mechanism, such as obtaining positive reward (+1) for successfully building an optical path that meets the optical path request, and obtaining negative reward (-1) for not meeting the optical path request, so that the DQN algorithm can intelligently explore the mapping relationship between the network state and the RSA decision, and continuously learn and optimize in the training process, thereby realizing intelligent solution to the RSA problem. The method proposed in the present application not only can adapt to the dynamic changes of network state, but also can improve the accuracy and robustness of RSA decision while maintaining the computing efficiency, and combines the consideration of spectrum fragmentation in the link, which can provide more efficient and intelligent resource allocation scheme for the optical network.

[0010] To achieve the above purposes, a DQN-based optical network routing and spectrum allocation algorithm comprises the following steps:

[0011] Step S1: defining the optical network environment and algorithm initialization, and constructing the basic framework of the DQN model;

[0012] Step S2: receiving the optical path request and executing the RSA decision;

[0013] Step S3: feeding back the corresponding reward according to the executed RSA decision, and training the DQN model;

[0014] Step S4: testing the DQN model, and selecting the optimal path and spectrum allocation scheme according to the current network state for each received optical path request.

[0015] As a further preferred technical solution of the above technical solution, step S1 is specifically implemented as the following steps:

[0016] Step S1.1: design a DQN model, including input layer, hidden layer and output layer; the input layer is used to receive information in the state space and convert it into a format for neural network processing; the hidden layer is used to extract state features and learn the mapping relationship between states and actions; the output layer is used to output the Q value corresponding to each candidate path;

[0017] Step S1.2: define the preferred state space in the RSA problem as the input of the DQN model; the preferred state space is an input feature vector composed of features extracted from the network topology connection, spectrum resource usage and light path request information; the parameters in the vector include One-Hot encoding corresponding to the source node-destination node pair in the light path request, bandwidth, index of candidate paths that meet the demand, total number of idle spectrum slots, number of spectrum slots that meet the demand, maximum size of idle spectrum slots that meet the demand, and minimum size of idle spectrum slots that meet the demand;

[0018] Step S1.3: define the preferred action space in the RSA problem, i.e. select a suitable path and spectrum allocation scheme; the DQN model outputs the Q value of all candidate paths according to the input feature vector of the state space, and selects the path according to the Q value, and then performs spectrum allocation.

[0019] As a further preferred technical solution of the above technical solution, for step S2,

[0020] Using the DQN model to make RSA decisions, first extract the features in the state space as the input feature vector of the DQN model, then the DQN model will output the Q value of all candidate paths, and according to the Q value, select the path and then perform spectrum allocation, specifically:

[0021] When allocating spectrum, first select the appropriate modulation format according to the transmission distance, get the corresponding number of spectrum slots that need to be allocated, and then select from the two methods of first fitting spectrum allocation and random spectrum allocation; for both methods, calculate the spectrum fragmentation degree caused by the two methods under the current state according to the formula A=(S+F cut +M align ) / C, and compare, finally adopt the method with smaller spectrum fragmentation value, so as to improve the utilization rate of spectrum resources in the network;

[0022] Wherein, S is the number of spectrum slots required to meet the light path request, F cut is the number of links that are originally continuous idle spectrum resources after allocating spectrum at the current position, M align is the sum of the number of idle spectrum slots in the same spectrum position in the adjacent link of the allocated link, and C is the number of all continuous idle spectrum slots that meet spectrum allocation in the current link.

[0023] As a further preferred technical solution of the above technical solution, for step S3, during the training process, the agent will select an action according to the current state, execute it to obtain a reward, and store the experience data into the experience replay Buffer, wait until the data in the replay Buffer is stored to a certain number, denoted as m, and then start training again, and after training, m / 2 of the earlier generated data in the replay Buffer will be deleted to keep the sample data in the replay Buffer updated; during the training process, whenever the number of data in the replay Buffer is stored to m again, the DQN model will randomly extract a batch of data from the experience replay Buffer for training and updating the parameters of the DQN model.

[0024] As a further preferred technical solution of the above technical solution, for step S4, for each new optical path request, the agent will select the optimal path and spectrum allocation scheme according to the current network state; then execute the scheme, update the network state, and record the parameters of each request including the allocation result, network request blocking rate, and spectrum utilization rate during the training process, through giving appropriate routing path selection and spectrum allocation strategy, reduce the blocking rate in the optical network, and improve the resource utilization rate.

[0025] In summary, the present application has the following technical effects:

[0026] (1) In the problem of complex routing and spectrum allocation in optical networks, the present application improves the problems of high blocking rate and low spectrum resource utilization rate in the traditional heuristic algorithm (such as Greedy algorithm, KPaths algorithm) applied in routing and spectrum allocation (RSA), and provides an efficient and adaptive solution for routing and spectrum allocation problem.

[0027] (2) The present application uses the reinforcement learning ability of DQN to intelligently select routing paths and consider the degree of spectrum fragmentation caused by the first fitting spectrum allocation and random selection spectrum allocation method when allocating spectrum resources, selects the method that causes less spectrum fragmentation, reduces the generation of spectrum fragmentation in the link, and improves the utilization rate of spectrum resources in the network.

[0028] (3) The present application improves and optimizes the frequency of training and target network update in spectrum allocation algorithm, and the update of exploration rate, not only improves the utilization rate of spectrum resources in the link, but also reduces the blocking rate of requests in the network under limited computing resources, thereby improving the overall performance index of the network in a limited time. In the later research content, the algorithm improvement for further reducing the degree of spectrum fragmentation in the network will be focused on, so as to reduce more request blocking rate. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a flow chart based on the DQN algorithm combined with routing and spectrum allocation;

[0030] Figure 2 is a schematic diagram of the DQN algorithm combined with RSA;

[0031] Figure 3 is a flow chart of the DQN algorithm combined with RSA;

[0032] Figure 4 is a schematic diagram of the training process of the main network and the target Q network;

[0033] Figure 5 is a partial screenshot of the Input_feature.txt file of the input feature vector during the training of the model;

[0034] Figure 6 is a partial screenshot of the DQN_feature.txt file of the data composition during the training process;

[0035] Figure 7 is a partial screenshot of the output of the data to the q_values.xlsx file during the training process;

[0036] Figure 8 is a comparison chart of the blocking rate of the traditional algorithm and the algorithm of the present application. DETAILED DESCRIPTION

[0037] The following description is provided to enable any person skilled in the art to practice the present application. The preferred embodiments in the following description are only examples of the present application and other obvious modifications can be made by those skilled in the art. The basic principles defined in the following description can be applied to other embodiments, modifications, improvements, equivalents and other technical solutions without departing from the spirit and scope of the present application.

[0038] In the preferred embodiments of the present application, those skilled in the art should note that the DQN and the like involved in the present application can be regarded as prior art.

[0039] Preferred embodiments.

[0040] The RSA problem usually refers to selecting a path from a set of predefined paths and allocating appropriate spectrum, and is more biased towards discrete problems. In reinforcement learning algorithms, Q-learning and DQN algorithms are both used to indirectly find the optimal strategy by estimating the value of each state or state-action pair, and are therefore more suitable for discrete action space. Other reinforcement learning methods, such as the Actor-Critic (AC) algorithm mentioned in the paper “A Multi-Agents Survivable Deep Reinforced Routing and Spectrum Assignment Algorithm in Elastic Optical Network”, directly search and optimize the policy itself, and usually output a specific action or probability distribution of actions, i.e. the selection of actions is a continuous variable, such as speed and angle, and is therefore more suitable for continuous action space. Moreover, the AC algorithm requires the training of two networks (Actor network and Critic network) simultaneously, and has a high algorithm complexity. The PPO (Proximal Policy Optimization) algorithm used in the paper “Deep-reinforcement-learning-based RMSCA for space division multiplexing networks with multi-core fibers” limits the magnitude of changes in the strategy during each update, thereby maintaining the stability of the training. This algorithm is simple to implement, but if applied to RSA, it may not be able to fully consider all factors when facing dynamic changes in network environment, such as network topology failures and spectrum resource changes, and therefore has limited applicability in practical applications.

[0041] Therefore, Q-learning and DQN algorithms may be more suitable for application in RSA, but when Q-learning algorithm is combined with RSA, it will face problems such as rapid increase in the size of Q-Table with the expansion of network size, slow convergence speed of the algorithm to obtain a better strategy, and the need for multiple iterations. The present application uses Deep Q-Network algorithm for routing and spectrum allocation (RSA) optimization.

[0042] The DQN-based optical network routing and spectrum allocation algorithm of the application comprises the following steps: constructing a DQN model in combination with an RSA problem, the model learning an optimal strategy of mapping an optical network state to an RSA strategy through a deep neural network; selecting a suitable candidate path according to an optical path request and a current network state, and fully considering a spectrum fragmentation factor when performing spectrum allocation; training the DQN model by using training data in an experience replay Buffer, the training target of the model being to maximize a reward function, and the reward function being designed according to whether the RSA decision successfully satisfies the optical path request and the like, so that the model can finally learn how to make an optimal RSA decision under different network states and realize efficient allocation of resources.

[0043] As shown in Figures 1-8 , the application discloses a DQN-based optical network routing and spectrum allocation algorithm, comprising the following steps:

[0044] Step S1: defining an optical network environment and algorithm initialization, and constructing a basic framework of a DQN model;

[0045] Step S2: receiving an optical path request, and performing an RSA (i.e. routing selection and spectrum allocation) decision;

[0046] Step S3: feeding back a corresponding reward according to the performed RSA decision, and training a DQN model;

[0047] Step S4: testing the DQN model, and selecting an optimal path and spectrum allocation scheme according to a current network state for each received optical path request.

[0048] Referring to Figure 2 , the combination of the Deep Q-Network algorithm and the routing and spectrum allocation method mainly comprises the following main parts:

[0049] 1 is an RSA environment, comprising an optical network environment and an optical path request, the optical network environment containing a network topology structure, link connection, spectrum usage of a link, and information such as a source node-destination node and bandwidth contained in the optical path request.

[0050] 2 is an action made by an agent, comprising outputting an action space by using a deep neural network, receiving reward feedback given by an environment, and randomly extracting data from an experience replay Buffer to train the network, so as to iteratively update network parameters.

[0051] 3 is that for each generated RSA strategy, four parameters of a corresponding state, action, reward and updated state are put into an experience replay Buffer, so as to facilitate subsequent training of a DQN model.

[0052] The specific principle process is: when the environment receives a light path request, the environment gives the agent the current environment state and the information contained in the light path request, the agent receives these information and then makes corresponding actions, that is, the agent and the environment interact to calculate the Q value of each candidate path, to select actions (path selection and spectrum allocation), the actions act on the environment to update the network state, and the data (state, action, reward, updated state) in these processes are stored in a data set (experience replay Buffer), and a batch of data is randomly selected from the data set to train the DQN network model during training; and the algorithm also establishes a target network for calculating the target Q value, and the parameters of the target network are copied from the main network at intervals. According to the training result, the parameters of the main network are updated to minimize the loss function (the average square difference between the Q value and the target Q value).

[0053] Specifically, step S1 is implemented as the following steps:

[0054] Step S1.1: design a DQN model including an input layer, a hidden layer and an output layer; the input layer is used to receive information in the state space and convert it into a format that can be processed by a neural network; the hidden layer is used to extract state features and learn the mapping relationship between states and actions; the output layer is used to output the Q value corresponding to each candidate path, which represents the expected reward that can be obtained by executing a certain action in the current state;

[0055] Step S1.2: define the preferred state space in the RSA problem as the input of the DQN model; the preferred state space is an input feature vector composed of features extracted from the network topology connection, spectrum resource usage and light path request information (etc.), which is composed of features extracted from the optical network environment and the light path request. The network environment includes the topology structure of the network, link connection, and spectrum usage of the link. The light path request contains information such as source node-destination node and bandwidth. The parameters in the vector include One-Hot encoding of the source node-destination node pair in the light path request (unselected topology nodes are 0 and selected nodes are 1), bandwidth, index of candidate paths that meet the demand, total number of free spectrum slots, number of spectrum slots that continuously meet the demand, maximum size of free spectrum slots that meet the demand, and minimum size of free spectrum slots that meet the demand;

[0056] Step S1.3: define the preferred action space in the RSA problem, that is, select a suitable path and spectrum allocation scheme; the DQN model outputs the Q value of all candidate paths according to the input feature vector of the state space (select the path according to the Q value combined with the ε-greedy strategy), and selects the path according to the Q value, and then performs spectrum allocation.

[0057] For step S1:

[0058] A: The optical network environment mainly includes defining the link state and traffic distribution of the network. The link state of the network includes the number of network nodes and link connections, etc. The link state of the optical network is stored in the dictionary linkmap, and the specific format is linkmap(a, b) = (n, c), where a is the source node, b is the destination node, c is the connection value, and n is a unique identifier for each pair of connections.

[0059] The traffic distribution of the network is stored in the random symmetric matrix trafic_dis. The matrix dimension is the number of network nodes. Each row of the matrix represents the traffic distribution of a node to other nodes. The symmetric matrix value(i, j) = value(j, i) is used because the traffic from i node to j node is the same as the traffic from j node to i node.

[0060] B: Algorithm initialization mainly includes defining the parameters required by the algorithm, including generating a candidate path list, loading the algorithm model, etc. First, the optical network environment and the parameters required by the DQN algorithm need to be defined, such as the number of optical network nodes and links, the dimension of the input feature vector of the DQN algorithm, the learning rate, etc.

[0061] Then generate the candidate path list: traverse all possible source node-destination node pairs, and add the corresponding traffic probability distribution to the prob_arr list, the corresponding node number to the Src_Dest_Pair list, and the node number on each path read to the Candidate_Paths.

[0062] Then set and initialize the DQN model and its training environment, judge whether the current model has been trained, if not, start training again, and reset the TensorFlow default graph. Define the algorithm model using the Adam optimizer to train and update the network weights, build the main network to generate the training target. Create multiple threads to train multiple agents in parallel with the environment, collect data and update network parameters, so as to reduce the training time, and finally define the saved trained model.

[0063] C: The basic framework of building the DQN algorithm model includes defining the structure of the deep neural network DNN, etc. The DNN network includes input layer, hidden layer and output layer. The input layer receives state features as input, maps the input features to neurons for processing through fully connected layers, and the output of this layer will be used as the input of the subsequent hidden layer.

[0064] The neurons of the hidden layer receive the outputs from the input layer, calculate the weighted sum of all input signals, and then calculate the output value through an activation function. ReLU is used as the activation function, which is used to introduce nonlinearity. The output of the hidden layer is passed to the next hidden layer.

[0065] The output layer generates the final prediction result, i.e., the expected Q value of all possible actions.

[0066] More specifically, for step S2,

[0067] The DQN model is used for routing selection and RSA decision-making. First, the features in the state space are extracted as the input feature vector of the DQN model. Then, the DQN model outputs the Q values of all candidate paths. After selecting the path according to the Q values (combined with the ε-greedy strategy), spectrum allocation is performed (the continuity constraint, adjacency constraint, and non-overlapping constraint of the spectrum in the network should be considered during spectrum allocation). When allocating spectrum, considering allocating spectrum at different positions may cause different degrees of spectrum fragmentation in the link. Specifically,

[0068] When allocating spectrum, the appropriate modulation format is first selected according to the transmission distance. After obtaining the number of spectrum slots that need to be allocated, two methods, i.e., the first fitting spectrum allocation method and the random spectrum allocation method, are selected. For both methods, the spectrum fragmentation degree caused by the two methods under the current state is calculated according to the formula A = (S + F cut + M align ) / C, and then compared. Finally, the method with smaller spectrum fragmentation value is adopted to improve the utilization rate of spectrum resources in the network.

[0069] wherein S is the number of spectrum slots required to meet the optical path request, F cut is the number of links that are originally continuous and idle spectrum resources that are cut into two pieces after allocating spectrum at the current position, M align is the sum of the number of spectrum slots that are idle at the same spectrum position in the adjacent links of the allocated link, and C is the number of all continuous idle spectrum slots that meet spectrum allocation in the current link.

[0070] For step S2:

[0071] A: First, define the functions needed when performing path routing and spectrum allocation. Including initialization functions, initializing parameters received by the agent, and parameters related to the TensorFlow graph, etc.; the function of copying the parameters of the Q value network to the target network; the path finding function, find the Kth path from the source node to the destination node, if the source node and the destination node are equal, an empty path will be output; the function of calculating the node information in the path, define p as a list of nodes contained in a path from the source node to the destination node, find the information of all adjacent node pairs in the p path in the linkmap, and then add it to the path node connection list; spectrum allocation function, traverse each link in the candidate path and each spectrum slot on the link, update the occupation of each spectrum slot by bitwise AND operation, if a spectrum slot is initially available but is occupied on a certain link, the final state of this spectrum slot should still be unavailable; query spectrum resource function, find whether there is continuous available spectrum resource in the given link spectrum, finally get the number of available spectrum resources, the list of available spectrum slots and the length of available spectrum slots.

[0072] And there is also a function to judge whether the spectrum allocation is successful, which is divided into two steps: first fitting allocation, that is, starting from the first allocable spectrum slot, if the condition is met, it can be allocated; query the idle spectrum slot in the path that meets the demand, and store the start and end positions of the continuous idle spectrum slot that meets the demand in the array. If there is spectrum resource that can be successfully allocated in the link, the start and end positions of the first fitting allocation spectrum and the array storing the start and end positions of all continuous idle spectrum slots that meet the demand in the path are finally obtained; update the spectrum resource occupation in the whole link function; release the spectrum resource occupied by the expired request function; calculate the path length function, finally get the total length of the whole path and the length of the link between nodes; calculate the number of spectrum slots required to meet the optical path request, when the modulation format of the signal is higher, the information carried by the same spectrum and the same bandwidth is more, and the same signal 16QAM and BPSK only needs one fourth of the bandwidth, at the same time, the higher the modulation format, the higher the requirement for the transmission quality of the signal, which can only be transmitted in a shorter distance. Therefore, when calculating, the required spectrum slot number is calculated according to the path length and bandwidth demand, that is, different signal modulation formats are adopted, 1 corresponds to BPSK, 2 corresponds to QPSK, 3 corresponds to 8QAM, and 4 corresponds to 16QAM.

[0073] B: Then receive the optical path request information, and make RSA decision according to the current network state and considering the impact of spectrum fragmentation when allocating spectrum.

[0074] By defining classes and methods, the generation of light path requests, the selection of paths, the allocation of spectrum, and the training of network models are implemented. First, some parameters need to be initialized, including the exploration rate, the number of training cycles, etc. Then, after entering the training cycle, in each cycle, the light path request is generated first, including randomly selecting the source node and the destination node of two points, and randomly generating an integer bandwidth demand between 25-100. And the One-Hot encoding corresponding to the node pair and the bandwidth are added to the input feature vector and the updated state feature vector.

[0075] Then, according to the source node-destination node pair, k candidate paths are selected from the candidate paths. Traverse the k candidate paths: first, judge whether the length of the candidate path is 0, if not, calculate the total length of the path, the length between nodes, the required spectrum number, the link connection contained in the establishment of the optical path connection, etc. Then judge whether there is a continuous empty spectrum in the path, only when there is and the empty spectrum meets the required spectrum number, it is satisfied. Put the index of the candidate path that meets the demand, the total number of idle spectrum slots, the number of continuous spectrum slots that meet the demand, the maximum size of the idle spectrum slot that meets the demand, and the minimum size of the idle spectrum slot that meets the demand into the input feature vector. Then convert the input feature vector into a numpy array, and use the reshape function to reshape the vector into a shape suitable for the input feature dimension of the model.

[0076] According to the input feature vector, the Q value of the k candidate paths is calculated. And adopt the ε-greedy strategy, that is, when the algorithm selects an action, a random number between 0 and 1 is generated first, if the random number is greater than ε, the action with the maximum Q value is selected and executed, if it is less than ε, a random action is selected and executed, and the initial setting of ε=1 can make the algorithm mainly in the exploration state in the early stage.

[0077] After selecting the path, the spectrum is allocated, and the impact of spectrum fragmentation is considered when allocating the spectrum. Among them, the spectrum fragmentation value corresponding to the allocation of different spectrum positions is calculated by using the following spectrum fragmentation function in the paper 《Fragmentation-Aware Routing, Modulation and Spectrum Assignment Algorithms in Elastic Optical Networks》.

[0078] A=(H*S+F cut +M align ) / C

[0079] Where H represents the number of candidate paths, S represents the number of spectrum slots required to meet the light path request, F cutM represents the number of links whose originally continuous idle spectrum resources are cut into two pieces after the spectrum is allocated at the current position align is the sum of the number of idle spectrum slots in the same position of the adjacent links of the allocated link. Because the algorithm in the present application is to allocate spectrum after determining which candidate path to select, the candidate path in the original formula is removed.

[0080] By comparing the A1 value obtained by first fitting the allocated spectrum with the A2 value obtained by randomly selecting a position to allocate from the idle spectrum resources of the link, the allocation method with the smaller value is finally selected to reduce the fragmentation of the spectrum in the link and improve the utilization of network resources.

[0081] Further, for step S3, during the training process, the agent, i.e. the executor and learner of the DQN model, will select an action according to the current state, execute it to obtain a reward (a positive reward (+1) will be given if a light path connection meeting the light path request is successfully established, otherwise a negative reward (-1) will be given), and store the experience data (state, action, reward, updated state) into the experience replay Buffer (buffer memory). When the data in the replay Buffer is stored to a certain amount, denoted as m, the training is started again, and after the training, m / 2 of the earlier generated data in the replay Buffer will be deleted to keep the sample data in the replay Buffer updated. During the training process, whenever the number of data in the replay Buffer is stored to m again, the DQN model will randomly extract a batch of data from the experience replay Buffer for training and updating the parameters of the DQN model (but in the early stage when the amount of data in the replay Buffer is small, the training will not be performed first). Moreover, the algorithm also combines the ε-greedy strategy when selecting a path, which will first generate a random number between 0 and 1. If the random number is greater than ε, the action with the maximum Q value will be selected for execution, and if it is less than ε, a random action will be selected for execution. The initial ε value set by the algorithm is 1, which is intended to let the algorithm mainly be in an exploration state in the early stage, and the ε value will decrease with the number of training, for example, without loss of generality, the ε value decreases by 1e-5 each time, and the lower limit decreases to 0.05. In this way, not only can the algorithm be prevented from falling into a local optimal solution, but also the adaptability of the algorithm can be improved.

[0082] For step S3:

[0083] A: According to the executed routing path selection and spectrum allocation strategy, if the light path connection can be successfully established, a positive reward (+1) is given, and the environment state of the entire network is updated, and the corresponding parameters are put into the updated state feature vector; otherwise, a blockage is generated, and a negative penalty (-1) is given.

[0084] B: Selecting not to train and update the network when the amount of data in the early experience replay Buffer is small, seeFigure 4 Only the current state, action, reward, updated state data are put into the experience replay Buffer; and the Q value of each action of the updated state is calculated by the target Q network, the maximum Q value is selected to calculate the target Q value, and the target Q value calculation formula is as follows.

[0085] Q(s,a)=R(s,a)+γ*max a′ Q(s′,a′)

[0086] Wherein Q(s,a) is the Q value of the current state s making action a; R(s,a) is the reward obtained by making action a for the current state s; max a′ Q(s′,a′) is the maximum Q value in the updated state s′; γ is a discount factor, used to balance the importance of future rewards in current decision-making, when the discount factor is larger, the algorithm will pay more attention to future rewards, and when the discount factor is smaller, the algorithm will pay more attention to the current immediate reward.

[0087] In the early stage, when the data in the replay Buffer is less, training is not performed first, when the data in the replay Buffer is accumulated enough, denoted as m, sample data are randomly extracted for training and updating network parameters, after training, m / 2 earlier generated data in the replay Buffer are deleted, and the target network parameters and v value are updated with the number of training times, the target network parameters are copied from the main network, and the ε value is reduced with the number of training times, for example, without loss of generality, the ε value is reduced by 1e-5 each time, and the lower limit is reduced to 0.05. And the parameters in each training process are output and saved, and the model parameters are saved every certain training time.

[0088] For step S4: finally, the loss function is calculated and the parameters of the network model are updated, wherein:

[0089] The loss function is calculated according to the average square difference between the Q value of the model output and the target Q value.

[0090]

[0091] Then the Adam optimizer is created to update the main network parameters to minimize the loss function, and the learning rate is used to control the speed of model parameter update.

[0092] Further, for step S4, for each new optical path request, the agent will select the optimal path and spectrum allocation scheme according to the current network state; then execute the scheme, update the network state, and record the parameters including the allocation result, network request blocking rate, and spectrum utilization rate of each request in the training process, by giving appropriate routing path selection and spectrum allocation strategy, reduce the blocking rate in the optical network, and improve the resource utilization rate.

[0093] See Figure 5 Input_feature.txt file composed of input state feature vectors of the model. Parameters include one-hot representation of source node and destination node in lightpath request, bandwidth size of 97, index 0 or 1 of candidate path, number of continuous spectrum slots meeting the requirements in the link, number of spectrum slots required to meet the bandwidth demand, maximum and minimum continuous spectrum slot size meeting the demand in the link.

[0094] See Figure 6 DQN_feature.txt file composed of data generated in the model training process. Parameters include total path length from source node to destination node of 1571, number of spectrum slots num_FS required to meet the lightpath request of 5, nodes [139, 170, 171] contained in the path from the source node to the destination node, path length span [4, 6, 11], spectrum start position, from the 0th position to the 4th position, corresponding to the following spectrum resource usage, and corresponding power spectral density.

[0095] See Figure 7 The data generated in the model training process is input into the q_values.xlsx file. The output parameters include ① (source node, destination node pair) corresponding to the storage in temp; ② bandwidth size current_bandwidth required by the lightpath request; ③ path index path_id (in the candidate path, the index value is 0 or 1); ④ total length of the selected path path_len; ⑤ link connection path_links contained in the establishment of the optical path connection; ⑥ path length lengths between nodes in the selected path; ⑦ modulation format modulation_format; ⑧ number of spectrum num_FS required to meet the lightpath request calculated according to the path length and bandwidth demand; ⑨ start position fs_start of the spectrum allocated on the selected link; ⑩ end position fs_end of the spectrum allocated on the selected link, if the values of fs_start and fs_end are both -1, it means that the spectrum resource on the selected link does not meet the lightpath request, and a blockage is generated; Spectrum utilization resource_util of the link; Q values q_values of the two candidate paths.

[0096] See Figure 8For the traditional algorithm and the blocking rate of the algorithm of the application comparison chart. The green line is the traditional heuristic (KSP) algorithm, the blue line is the blocking rate of the traditional DQN algorithm without considering spectrum fragmentation, and the orange line is the blocking rate of the algorithm of the application considering spectrum fragmentation. From the image, it can be seen that the blocking rate of the routing and spectrum allocation algorithm combined with the DQN algorithm is greatly improved compared with the traditional algorithm. Although the blocking rates of the traditional DQN and the algorithm of the application are not much different in the early stage, the number of optical path requests is small, and the idle spectrum resources in the network link are relatively continuous, so considering the spectrum fragmentation has little effect on the performance improvement of the algorithm. But in the later stage, when the number of optical path requests is large, the idle spectrum resources of the link are relatively dispersed, and considering the degree of spectrum fragmentation can reduce the blocking rate of the request in the network. When the number of optical path requests reaches 3000, the blocking rate of the traditional DQN algorithm without considering spectrum fragmentation is about 0.13, while the blocking rate of the algorithm of the application is only about 0.1, which can realize about 20% reduction in blocking.

[0097] It is worth mentioning that the DQN and other technical features involved in the present patent application should be regarded as prior art. The specific structure, working principle and possible control method and spatial arrangement method of these technical features can be selected conventionally in the art, and should not be regarded as the invention point of the present patent. The present patent will not be further expanded and detailed.

[0098] For those skilled in the art, the technical solutions described in the foregoing embodiments can still be modified or some technical features can be replaced by equivalents. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A DQN-based optical network routing and spectrum allocation algorithm, characterized in that, Comprising the following steps: Step S1: defining the optical network environment and algorithm initialization, constructing the basic framework of the DQN model; Step S2: receiving an optical path request and executing RSA decision; Using the DQN model to make RSA decisions, first extracting the features in the state space as the input feature vector of the DQN model, then the DQN model will output the Q values of all candidate paths, and according to the Q values, the path is selected, and then the spectrum is allocated, specifically: When allocating spectrum, the modulation format is selected according to the transmission distance first, and then the number of spectrum slots corresponding to the required allocation is obtained. After that, the spectrum is allocated from two methods of initial fitting and random selection. For both methods, the spectrum fragmentation degree caused by the current state is calculated according to the formula The spectrum fragmentation degrees caused by the two methods are compared, and the method with smaller spectrum fragmentation value is finally adopted, so as to improve the utilization rate of spectrum resources in the network. in, The number of spectrum slots required to satisfy the lightpath request, The number of links that will be split into two blocks after allocating spectrum at the current location. is the sum of the number of idle spectrum slots at the same spectrum position in the adjacent links of the allocated link, The number of all consecutive idle spectrum slots that meet spectrum allocation requirements in the current link; Step S3: according to the feedback of the executed RSA decision, the corresponding reward is fed back, and the DQN model is trained; Step S4: test the DQN model, for each received optical path request, select the optimal path and spectrum allocation scheme according to the current network state.

2. The DQN-based optical network routing and spectrum allocation algorithm according to claim 1, wherein, For step S1, the following steps are implemented: Step S1.1: design a DQN model, including input layer, hidden layer and output layer; the input layer is used to receive information in the state space and convert it into a format that can be processed by the neural network; the hidden layer is used to extract state features and learn the mapping relationship between states and actions; the output layer is used to output the Q value corresponding to each candidate path; Step S1.2: define the preferred state space in the RSA problem as the input of the DQN model; the preferred state space is the input feature vector composed of the features extracted from the network topology connection, spectrum resource usage and optical path request information. The parameters in the vector include the One-Hot encoding of the source node-destination node pair in the optical path request, the bandwidth, the index of the candidate path that meets the demand, the total number of idle spectrum slots, the number of spectrum slots that meet the demand, the maximum size of the idle spectrum slot that meets the demand, and the minimum size of the idle spectrum slot that meets the demand; Step S1.3: define the preferred action space in the RSA problem, i.e. select the appropriate path and spectrum allocation scheme; the DQN model outputs the Q values of all candidate paths according to the input feature vector of the state space, and selects the path according to the Q values, and then performs spectrum allocation.

3. The DQN-based optical network routing and spectrum allocation algorithm according to claim 2, wherein, For step S3, during the training process, the agent will select an action according to the current state, execute it to get a reward, and store the experience data in the experience replay Buffer. When the data in the replay Buffer is stored to a certain number, denoted as m, the training begins again, and after training, m / 2 of the earlier generated data in the replay Buffer is deleted to keep the sample data in the replay Buffer updated. During the training process, whenever the number of data in the replay Buffer is stored again to m, the DQN model will randomly extract a batch of data from the experience replay Buffer for training and updating the parameters of the DQN model.

4. The DQN-based optical network routing and spectrum allocation algorithm according to claim 3, wherein, For step S4, for each new optical path request, the agent will select the optimal path and spectrum allocation scheme according to the current network state; then execute the scheme, update the network state, and record the parameters including allocation results, network request blocking rate, and spectrum utilization rate for each request during the training process. By giving appropriate routing path selection and spectrum allocation strategies, the blocking rate in the optical network is reduced and the resource utilization rate is improved.

Images