Optical network routing and spectrum allocation algorithm based on DQN

Abstract

The invention discloses an optical network routing and spectrum allocation algorithm based on a DQN, and the algorithm comprises the steps: S1, defining an optical network environment, initializing the algorithm, and constructing a basic framework of a DQN model; s2, receiving an optical path request, and executing an RSA decision; s3, feeding back a corresponding reward according to the executed RSA decision, and training the DQN model; and S4, testing the DQN model, and selecting an optimal path and spectrum allocation scheme for the optical path request received each time according to the current network state. The optical network routing and spectrum allocation algorithm based on the DQN disclosed by the invention optimizes and improves the spectrum allocation, the frequency of training and target network updating in the algorithm, the updating of the exploration rate and the like, and can improve the problems of high blocking rate, low spectrum resource utilization rate and the like in the traditional heuristic algorithm.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical network resource configuration, and in particular relates to an optical network routing and spectrum allocation (Routing and Spectrum Allocation, RSA) algorithm based on DQN (Deep Q-Network). Background Art

[0002] With the rapid development of information technology, optical network as a new type of optical network technology has become an indispensable part of modern society. However, with the expansion of network scale and the increase of user demand, how to efficiently perform routing and spectrum allocation has become a key issue to be solved in the field of optical network.

[0003] The paper "Online routing and spectrum allocation in elastic optical networks based on dueling Deep Q-network" proposed that RSA means to establish an optical path for each business demand by selecting appropriate routes and allocating sufficient spectrum, that is, to establish an optical path connection between the source node and the destination node, and to allocate an appropriate number of contiguous, continuous, and non-overlapping spectrum slots (FSs). Among them, contiguous refers to the continuity constraint, requiring that the frequency band allocated to a business demand must be composed of continuous FSs; continuous refers to the continuity constraint, requiring that continuous FSs with the same position index must be allocated on each link of a business demand path; non-overlapping refers to the non-overlapping constraint, that is, an occupied FS cannot be allocated to another traffic demand.

[0004] In optical networks, the formulation of RSA decisions depends on comprehensive consideration of network status, traffic demand, and available spectrum resources, which makes RSA decision-making difficult. In addition, because the solution space of the RSA problem is huge, the traditional exhaustive method will be unable to cope with large-scale networks and it is difficult to give the optimal solution within a reasonable time; the traditional heuristic algorithms introduced in the paper "Study and Analysis of Routing and Spectrum Allocation (RSA) and Routing, Modulation and Spectrum Allocation (RMSA) Algorithms in Elastic Optical Networks (EONs)", such as Greedy algorithm and KPaths algorithm, are prone to fall into local optimal solutions and cannot find the global optimal solution, which will lead to problems such as low spectrum utilization, thereby affecting the overall performance of the network; and traditional methods are often difficult to adapt to complex and changing network environments. Therefore, a method that can intelligently adapt to network changes and efficiently solve RSA problems is needed.

[0005] The paper "Deep reinforcement learning for comprehensive route optimization in elastic optical networks using generative strategies" uses a reinforcement learning algorithm and combines it with a generative strategy (GS) to narrow the search range of the action space and improve the efficiency and accuracy of decision-making. However, the introduction of the generative strategy also increases the computational complexity of the model, and the GS model cannot adapt to more complex problems as the author expected. The paper "Online routing and spectrum allocation in elastic optical networks based on dueling deep q-network" uses the Dueling network architecture in the reinforcement learning method, decomposing the Q value into state value and action advantage, which helps the network learn and take actions faster and improves learning efficiency, but the training complexity of the model will also increase accordingly.

[0006] Therefore, further improvements are made to the above problems. Summary of the invention

[0007] The main purpose of the present invention is to provide an optical network routing and spectrum allocation algorithm based on DQN, which optimizes and improves the spectrum allocation, the frequency of training and target network update in the algorithm, and the update of the exploration rate, etc., and can improve the problems of high blocking rate and low spectrum resource utilization existing in traditional heuristic algorithms (such as Greedy algorithm and KPaths algorithm).

[0008] Another object of the present invention is to provide an optical network routing and spectrum allocation algorithm based on DQN, which combines the reinforcement learning algorithm with the RSA problem in the elastic optical network, and can be used as an effective method for optimizing the RSA problem. Therefore, the present invention adopts the DQN algorithm in the reinforcement learning method, and makes optimizations in other aspects such as considering the impact of spectrum fragmentation on the basis of the traditional algorithm.

[0009] The traditional DQN algorithm uses a deep neural network to extract features and approximate the value function of complex network states. The present invention defines an optimal state space, such as the source node-destination node, bandwidth information, link and spectrum resource information in the optical network environment, etc. contained in the optical path request; an action space, such as selecting a certain path, allocating a certain spectrum, etc.; a reward mechanism, such as a positive reward (+1) is obtained if the optical path request is met and the optical path is successfully built, and a negative reward (-1) is obtained if it is not met, so that the DQN algorithm can intelligently explore the mapping relationship between the network state and the RSA decision, and continuously learn and optimize during the training process, thereby realizing the intelligent solution of the RSA problem. The method proposed in the present invention can not only adapt to the dynamic changes of the network state, but also improve the accuracy and robustness of the RSA decision while maintaining the computational efficiency, and combined with the consideration of spectrum fragmentation in the link, it can provide a more efficient and intelligent resource allocation solution for the optical network.

[0010] To achieve the above objectives, an optical network routing and spectrum allocation algorithm based on DQN includes the following steps:

[0011] Step S1: define the optical network environment and algorithm initialization, and build the basic framework of the DQN model;

[0012] Step S2: receiving a light path request and executing RSA decision;

[0013] Step S3: Feedback the corresponding reward according to the executed RSA decision and train the DQN model;

[0014] Step S4: Test the DQN model and select the optimal path and spectrum allocation scheme for each received lightpath request according to the current network status.

[0015] As a further preferred technical solution of the above technical solution, step S1 is specifically implemented as follows:

[0016] Step S1.1: Design a DQN model, including an input layer, a hidden layer, and an output layer; the input layer is used to receive information in the state space and convert it into a format for neural network processing; the hidden layer is used to extract state features and learn the mapping relationship between state and action; the output layer is used to output the Q value corresponding to each candidate path;

[0017] Step S1.2: define the preferred state space in the RSA problem as the input of the DQN model; the preferred state space is the input feature vector composed of features extracted from the network topology connection, spectrum resource usage and lightpath request information, and the parameters in the vector include the One-Hot encoding corresponding to the source node-destination node in the lightpath request, bandwidth, index of the candidate path that meets the demand, the total number of idle spectrum slots, the number of spectrum slots that meet the demand continuously, the maximum size of the idle spectrum slots that meet the demand and the minimum size of the idle spectrum slots that meet the demand;

[0018] Step S1.3: Define the preferred action space in the RSA problem, that is, select the appropriate path and spectrum allocation scheme; the DQN model outputs the Q value of all candidate paths according to the input feature vector of the state space, selects the path according to the Q value, and then performs spectrum allocation.

[0019] As a further preferred technical solution of the above technical solution, for step S2,

[0020] Using the DQN model for RSA decision-making, first extract the features in the state space as the input feature vector of the DQN model. Then the DQN model outputs the Q values ​​of all candidate paths and selects a path based on the Q value before allocating spectrum. Specifically,

[0021] When allocating spectrum, the appropriate modulation format is first selected according to the transmission distance. After obtaining the number of spectrum slots to be allocated, the spectrum is allocated by first fitting and randomly selecting. For both methods, the formula A = (S + F cut +M align ) / C calculates the spectrum fragmentation degrees caused by the two methods in the current state, and compares them. Finally, the method with the smaller spectrum fragmentation value is adopted to improve the utilization rate of spectrum resources in the network;

[0022] Where S is the number of spectrum slots required to satisfy the lightpath request, F cut M is the number of links that will be split into two pieces after allocating spectrum at the current location. align is the sum of the number of idle spectrum slots at the same spectrum position in the adjacent links of the allocated link, and C is the number of all continuous idle spectrum slots that meet the spectrum allocation in the current link.

[0023] As a further preferred technical solution of the above technical solution, for step S3, during the training process, the intelligent body selects actions according to the current state, obtains rewards after execution, and stores the experience data in the experience replay buffer. When a certain number of data in the replay buffer is stored, denoted as m, training will be started again, and after training, m / 2 earlier generated data in the replay buffer will be deleted to keep the sample data in the replay buffer updated; during the training process, whenever the number of data in the replay buffer is stored to m again, the DQN model will randomly extract a batch of data from the experience replay buffer for training and updating the parameters of the DQN model.

[0024] As a further preferred technical solution of the above technical solution, for step S4, for each new optical path request, the intelligent body will select the optimal path and spectrum allocation plan according to the current network status; then execute the plan, update the network status, and record the parameters of each request including allocation results, network request blocking rate, and spectrum utilization during the training process. By giving appropriate routing path selection and spectrum allocation strategies, the blocking rate in the optical network can be reduced and resource utilization can be improved.

[0025] In summary, the present invention has the following technical effects:

[0026] (1) In the complex routing and spectrum allocation problems of optical networks, the present invention improves the problems of high blocking rate and low spectrum resource utilization existing in traditional heuristic algorithms (such as Greedy algorithm and KPaths algorithm) used in routing and spectrum allocation (RSA), and provides an efficient and adaptive solution to the routing and spectrum allocation problems.

[0027] (2) The present invention utilizes the reinforcement learning capability of DQN to intelligently select routing paths and considers the application of two methods when allocating spectrum resources: first fitting spectrum allocation and random selection of spectrum allocation methods. The method that causes less spectrum fragmentation is selected for application, thereby reducing the generation of spectrum fragments in the link and improving the utilization rate of spectrum resources in the network.

[0028] (3) The present invention combines DQN with routing and spectrum allocation algorithms to improve and optimize spectrum allocation, the frequency of training and target network updates in the algorithm, and the update of the exploration rate. This not only improves the utilization of spectrum resources in the link, but also reduces the blocking rate of requests in the network under limited computing resources, thereby improving the overall performance indicators of the network within a limited time. In the later research content, we will focus on algorithm improvements that further reduce the degree of spectrum fragmentation in the network, thereby reducing more request blocking rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart based on the combination of DQN algorithm with routing and spectrum allocation;

[0030] Figure 2 This is the schematic diagram of the combination of DQN algorithm and RSA;

[0031] Figure 3 It is a flowchart combining the DQN algorithm with RSA;

[0032] Figure 4 This is a schematic diagram of the process of training the main network and the target Q network;

[0033] Figure 5 This is a partial screenshot of the Input_feature.txt file that inputs the feature vector when training the model;

[0034] Figure 6 This is a partial screenshot of the DQN_feature.txt file that contains the data during training.

[0035] Figure 7 This is a partial screenshot of the data output to the q_values.xlsx file during the training process;

[0036] Figure 8 This is a comparison chart of the blocking rates of the traditional algorithm and the algorithm of the present invention. DETAILED DESCRIPTION

[0037] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0038] In the preferred embodiments of the present invention, those skilled in the art should note that the DQN and the like involved in the present invention may be regarded as prior art.

[0039] Preferred embodiments.

[0040] The RSA problem usually refers to selecting a path from a set of predefined paths and allocating an appropriate spectrum, which is more inclined to a discrete problem. In reinforcement learning algorithms, both Q-learning and DQN algorithms indirectly find the optimal strategy by estimating the value of each state or state-action pair, so they are more suitable for discrete action spaces. Other reinforcement learning methods, such as the Actor-Critic (AC) algorithm mentioned in the paper "A Multi-Agents Survivable Deep Reinforced Routing and SpectrumAssignment Algorithmin Elastic Optical Network", directly search and optimize the strategy itself, usually outputting a specific action or the probability distribution of the action, that is, the choice of action is a continuous variable, such as movement speed, angle, etc., so it is more suitable for continuous action space, and the AC algorithm needs to train two networks (Actor network and Critic network) at the same time, and the algorithm complexity is relatively high; the PPO (Proximal Policy Optimization) algorithm used in the paper "Deep-reinforcement-learning-based RMSCA for space division multiplexing networkswith multi-core fibers" maintains the stability of training by limiting the amplitude of change each time the strategy is updated. The algorithm is simple to implement, but if applied to RSA, it may not be able to fully consider all factors when facing dynamic changes in the network environment, such as network topology failures, spectrum resource changes, etc., so it is not very applicable in practical applications.

[0041] Therefore, Q-learning and DQN algorithms may be more suitable for application in RSA, but when Q-learning algorithm is combined with RSA, it will face the problems that as the network scale expands, the size of Q-Table will increase rapidly, the algorithm needs to go through multiple iterations to obtain a better strategy, and the convergence speed is slow. The present invention adopts Deep Q-Network algorithm for routing and spectrum allocation (RSA) optimization.

[0042] The DQN-based optical network routing and spectrum allocation algorithm of the present invention includes: building a DQN model in combination with the RSA problem, and the model learns the optimal strategy of mapping the optical network state to the RSA strategy through a deep neural network. The algorithm selects a suitable candidate path according to the optical path request and the current network state, and fully considers the factor of spectrum fragmentation when allocating the spectrum; the DQN model is trained using the training data in the experience playback buffer, and the training goal of the model is to maximize the reward function, and the reward function is designed according to indicators such as whether the RSA decision successfully satisfies the optical path request, so that the model can finally learn how to make the optimal RSA decision under different network states and realize efficient allocation of resources.

[0043] like Figure 1-8 As shown, the present invention discloses an optical network routing and spectrum allocation algorithm based on DQN, comprising the following steps:

[0044] Step S1: define the optical network environment and algorithm initialization, and build the basic framework of the DQN model;

[0045] Step S2: receiving a lightpath request and executing RSA (i.e., routing selection and spectrum allocation) decision;

[0046] Step S3: Feedback the corresponding reward according to the executed RSA decision and train the DQN model;

[0047] Step S4: Test the DQN model and select the optimal path and spectrum allocation scheme for each received lightpath request according to the current network status.

[0048] See also Figure 2 ,The Deep Q-Network algorithm combined with the routing and spectrum allocation methods mainly includes the following main parts:

[0049] 1 is the RSA environment, including the optical network environment and the optical path request. The optical network environment includes the network topology, link connections, link spectrum usage, and information such as the source node-destination node and bandwidth included in the optical path request.

[0050] 2 is the action taken by the intelligent agent, including using the deep neural network to output the action space, receiving reward feedback from the environment, and randomly extracting data from the experience replay buffer to train the network to iteratively update the network parameters.

[0051] 3 For each generated RSA strategy, the corresponding state, action, reward, and updated state parameters are placed in the experience playback buffer for subsequent training of the DQN model.

[0052] The specific principle process is as follows: when the environment receives a light path request, the environment gives the agent the current state of the environment and the information contained in the light path request. The agent receives this information and then takes corresponding actions, that is, the agent interacts with the environment to calculate the corresponding Q value of each candidate path to select an action (path selection and spectrum allocation). The action acts on the environment to update the network state, and the data in these processes (state, action, reward, updated state) are stored in a data set (experience playback buffer). During training, a batch of data will be randomly selected from the data set to train the DQN network model; and the algorithm also establishes a target network for calculating the target Q value, and the parameters of the target network are copied from the main network at regular intervals. The parameters of the main network are updated according to the training results to minimize the loss function (the average square difference between the Q value and the target Q value).

[0053] Specifically, step S1 is implemented as follows:

[0054] Step S1.1: Design a DQN model, including an input layer, a hidden layer, and an output layer; the input layer is used to receive information in the state space and convert it into a format that can be processed by the neural network; the hidden layer is used to extract state features and learn the mapping relationship between state and action; the output layer is used to output the Q value corresponding to each candidate path, which represents the expected reward that may be obtained by performing a certain action in the current state;

[0055] Step S1.2: define the preferred state space in the RSA problem as the input of the DQN model; the preferred state space is the input feature vector composed of features extracted from the network topology connection, spectrum resource usage and optical path request information (etc.) (the preferred state space is extracted from the optical network environment and the optical path request to form the feature vector, the optical network environment includes the network topology, link connection, link spectrum usage, source node-destination node and bandwidth information contained in the optical path request), and the parameters in the vector include the One-Hot code corresponding to the source node-destination node in the optical path request (the unselected topological node is 0, the selected one is 1), bandwidth, index of candidate paths that meet the requirements, total number of idle spectrum slots, number of spectrum slots that meet the requirements continuously, maximum size of idle spectrum slots that meet the requirements and minimum size of idle spectrum slots that meet the requirements;

[0056] Step S1.3: Define the preferred action space in the RSA problem, that is, select the appropriate path and spectrum allocation scheme; the DQN model outputs the Q value of all candidate paths according to the input feature vector of the state space (select the path according to the Q value combined with the ε-greedy strategy), and selects the path according to the Q value, and then performs spectrum allocation.

[0057] For step S1:

[0058] A: The optical network environment mainly includes the definition of the link state and traffic distribution of the network. The link state of the network includes the number of network nodes and link connections. The link state of the optical network is stored in the definition dictionary linkmap. The specific format is linkmap(a,b)=(n,c), where a is the source node, b is the destination node, c is the connection value, and n is the unique identifier for each pair of connections.

[0059] Define a random symmetric matrix trafic_dis to store the traffic distribution of the network. The matrix dimension is the number of network nodes. Each row of the matrix represents the traffic distribution from a node to other nodes. The symmetric matrix value(i,j)=value(j,i) is used because we hope that the traffic from node i to node j is the same as the traffic from node j to node i.

[0060] B: Algorithm initialization mainly includes defining the parameters required by the algorithm, including generating a list of candidate paths, loading the algorithm model, etc. First, it is necessary to define the optical network environment and the parameters required by the DQN algorithm, such as the number of optical network nodes and links, the dimension of the DQN algorithm input feature vector, and the learning rate.

[0061] Then generate a list of candidate paths: by traversing all possible source node-destination node pairs, adding the corresponding traffic probability distribution to the prob_arr list, adding the corresponding node number to the Src_Dest_Pair list, and storing the node number on each path read into Candidate_Paths.

[0062] Then set and initialize the DQN model and its training environment, determine whether the trained model is currently loaded, if not, restart the training, and reset the TensorFlow default graph. Define the algorithm model to use the Adam optimizer for training to update the network weights, and build the main network to generate training targets. Create multi-threading, train multiple agents to interact with the environment in parallel, collect data and update network parameters, so as to reduce training time, and finally define and save the trained model.

[0063] C: The basic framework for building a DQN algorithm model includes defining the deep neural network DNN structure. The DNN network consists of an input layer, a hidden layer, and an output layer. The input layer receives state features as input, maps the input features to neurons for processing through a fully connected layer, and uses the output of this layer as the input of the subsequent hidden layer.

[0064] The neurons in the hidden layer receive the output from the input layer, calculate the weighted sum of all input signals, and then calculate the output value through the activation function. ReLU is used as the activation function. The role of the activation function is to introduce nonlinearity. The output of the hidden layer will be passed to the next hidden layer.

[0065] The output layer generates the final predictions, ultimately producing the expected Q-values ​​for all possible actions.

[0066] More specifically, for step S2,

[0067] The DQN model is used for routing selection and RSA decision-making. First, the features in the state space are extracted as the input feature vector of the DQN model. Then the DQN model outputs the Q values ​​of all candidate paths and selects the path based on the Q value (combined with the ε-greedy strategy). Then, spectrum allocation is performed (the continuity constraint, proximity constraint, and non-overlap constraint of the spectrum in the network should be considered when allocating spectrum). When allocating spectrum, it is considered that allocating spectrum at different locations will cause different degrees of spectrum fragmentation in the link. Specifically:

[0068] When allocating spectrum, the appropriate modulation format is first selected according to the transmission distance. After obtaining the number of spectrum slots to be allocated, the spectrum is allocated by first fitting and randomly selecting. For both methods, the formula A = (S + F cut +M align ) / C calculates the spectrum fragmentation degrees caused by the two methods in the current state, and compares them. Finally, the method with the smaller spectrum fragmentation value is adopted to improve the utilization rate of spectrum resources in the network;

[0069] Where S is the number of spectrum slots required to satisfy the lightpath request, F cut M is the number of links that will be split into two pieces after allocating spectrum at the current location. align is the sum of the number of idle spectrum slots at the same spectrum position in the adjacent links of the allocated link, and C is the number of all continuous idle spectrum slots that meet the spectrum allocation in the current link.

[0070] For step S2:

[0071] A: First, define the functions required for path routing selection and spectrum allocation. Including the initialization function, which initializes the parameters received by the agent and the parameters related to the TensorFlow graph, etc.; the function that copies the parameters of the Q value network to the target network; the path search function, which searches for the Kth path from the source node to the destination node. If the source node and the destination node are equal, an empty path will be output; the function that calculates the node information in the path, defines p as a list of nodes from the source node to the destination node, searches for the information of all adjacent node pairs in the p path in the linkmap, and then adds them to the path node connection list; the spectrum allocation function, which traverses each link in the candidate path and the spectrum slot on each link, and updates the occupancy of each spectrum slot by bitwise AND operation. If the initial state of a spectrum slot is available, but it is occupied on a certain link, the final state of this spectrum slot should still be unavailable; the spectrum resource query function, which searches for whether there are continuous available spectrum resources in the spectrum of a given link, and finally obtains the number of available spectrum resources, the list of available spectrum slots, and the length of available spectrum slots.

[0072] There is also a function to determine whether the spectrum allocation is successful, which is divided into two steps: the first fitting allocation, that is, starting from the first allocable spectrum slot, the query can be allocated if the conditions are met; query the idle spectrum slots that meet the requirements in the path, and store the starting and ending positions of the continuous idle spectrum slots that meet the requirements in the array. If there are spectrum resources that can be successfully allocated in the link, the starting and ending positions of the first fitting allocation spectrum are finally obtained, as well as the array storing the starting and ending positions of all the continuous idle spectrum slots that meet the requirements in the path; update the spectrum resource occupancy status function in the entire link; release the spectrum resource occupied by expired requests; calculate the path length function, and finally get the total length of the entire path, as well as the length of the link between each node; calculate the number of spectrum slots required to meet the optical path request. When the modulation format of the signal is higher, the same spectrum and the same bandwidth carry more information. The same signal 16QAM only needs one-fourth of the bandwidth compared to BPSK. At the same time, the higher the modulation format, the higher the transmission quality requirements for the signal, and it can only be transmitted over a shorter distance. Therefore, the required number of spectrum slots is calculated based on the path length and bandwidth requirements, that is, different signal modulation formats are used, 1 corresponds to BPSK, 2 corresponds to QPSK, 3 corresponds to 8QAM, and 4 corresponds to 16QAM.

[0073] B: Then it receives the lightpath request information and makes RSA decisions based on the current network status and considering the impact of spectrum fragmentation when allocating spectrum.

[0074] By defining classes and methods, functions such as lightpath request generation, path selection, spectrum allocation, and network model training are implemented. First, some parameters need to be initialized, including exploration rate, number of training cycles, etc. Then, after entering the training cycle, a lightpath request will be generated in each cycle, including randomly selecting two source nodes and destination nodes, and randomly generating integer bandwidth requirements between 25-100). The One-Hot encoding and bandwidth corresponding to the node are added to the input feature vector and the updated state feature vector.

[0075] Then, k candidate paths are selected from the candidate paths according to the source node-destination node pair, and the k candidate paths are traversed: first, determine whether the length of the candidate path is 0. If not, calculate the total length of the path, the length between each node, the number of spectrums required, the link connection parameters included in the establishment of the optical path connection, and then determine whether there is a continuous free spectrum in the path. Only when it exists and the free spectrum meets the required number of spectrums, the demand is met. The five parameters of the index of the candidate path that meets the demand, the total number of free spectrum slots, the number of continuous spectrum slots that meet the demand, the maximum size of the free spectrum slots that meet the demand, and the minimum size of the free spectrum slots that meet the demand are put into the input feature vector. Then convert the input feature vector into a numpy array, and then use the reshape function to reshape the vector into a shape suitable for the model input feature dimension.

[0076] The Q values ​​of k candidate paths are calculated based on the input feature vector. The ε-greedy strategy is adopted, that is, when the algorithm selects an action, it will first generate a random number between 0 and 1. If the random number is greater than ε, the action with the largest Q value is selected for execution. If it is less than ε, an action is randomly selected for execution. The initial setting of ε = 1 can keep the algorithm mainly in an exploration state in the early stage.

[0077] After selecting the path, spectrum allocation is performed, and the impact of spectrum fragmentation is considered when allocating spectrum. The following spectrum fragmentation function in the paper "Fragmentation-Aware Routing, Modulation and Spectrum Assignment Algorithms in Elastic Optical Networks" is used to calculate the spectrum fragmentation values ​​corresponding to different spectrum positions.

[0078] A=(H*S+F cut +M align ) / C

[0079] Where H represents the number of candidate paths, S represents the number of spectrum slots required to satisfy the lightpath request, and F cutIndicates the number of links that will be split into two pieces after allocating spectrum at the current position. align It refers to the sum of the number of free spectrum slots at the same position in the adjacent links of the allocated link. Because the algorithm in the present invention allocates spectrum after determining which candidate path to select, the candidate path in the original formula is removed.

[0080] The A1 value obtained by calculating the first fitting allocation spectrum is compared with the A2 value obtained by randomly selecting a position from the link's idle spectrum resources. Finally, the allocation method with a smaller value is selected to reduce spectrum fragmentation in the link and improve the utilization of network resources.

[0081] Furthermore, for step S3, during the training process, the agent, i.e., the executor and learner of the DQN model, will select actions according to the current state, and will be rewarded after execution (a positive reward (+1) will be given if a light path connection that satisfies the light path request is successfully established, otherwise a negative reward (-1) will be given), and the experience data (state, action, reward, updated state) will be stored in the experience replay Buffer (buffer memory). When a certain amount of data is stored in the replay Buffer, denoted as m, training will be started again, and after training, m / 2 earlier generated data in the replay Buffer will be deleted to keep the sample data in the replay Buffer updated; during the training process, whenever the amount of data in the replay Buffer is stored to m again, the DQN model will randomly extract a batch of data from the experience replay Buffer for training and updating the parameters of the DQN model (but when the amount of data in the early replay Buffer is small, training will not be performed first). The algorithm also combines the ε-greedy strategy when selecting a path. When selecting a path, it will first generate a random number between 0 and 1. If the random number is greater than ε, the action with the largest Q value will be selected for execution. If it is less than ε, an action will be randomly selected for execution. The algorithm initially sets ε=1 to keep the algorithm in an exploratory state in the early stage, and the ε value will decrease with the number of training times. For example, without loss of generality, the ε value is reduced by 1e-5 each time, and the lower limit is reduced to 0.05. This not only prevents the algorithm from falling into a local optimal solution, but also improves the adaptability of the algorithm.

[0082] For step S3:

[0083] A: According to the executed routing path selection and spectrum allocation strategy, if the optical path connection can be successfully established, a positive reward (+1) is given, and the environmental state of the entire network is updated, and the corresponding parameters are placed in the updated state feature vector; otherwise, congestion occurs and a negative penalty (-1) is given.

[0084] B: Choose not to train and update the network when there is less data in the early experience playback buffer, see Figure 4 , only the current state, action, reward, and updated state data are put into the experience replay buffer; and the Q value of each action in the updated state is calculated through the target Q network, and the target Q value is calculated by selecting the one with the largest Q value. The target Q value calculation formula is as follows.

[0085] Q(s,a)=R(s,a)+γ*max a′ Q(s′,a′)

[0086] Where Q(s,a) is the Q value of taking action a in the current state s; R(s,a) is the reward for taking action a in the current state s; max a′ Q(s′,a′) is the maximum Q value in the updated state s′; γ is the discount factor, which is used to balance the importance of future rewards in current decisions. When the discount factor is large, the algorithm will pay more attention to future rewards. When the discount factor is small, the algorithm will pay more attention to current immediate rewards.

[0087] In the early stage, when there is less data in the playback buffer, we choose not to train first. When the data in the playback buffer is accumulated enough, denoted as m, we randomly extract sample data for training and update network parameters. After training, we delete m / 2 earlier data in the playback buffer, and update the target network parameters and v value according to the number of training times. The update of the target network parameters is to copy the parameters of the main network, and the ε value will decrease with the number of training times. For example, without loss of generality, the ε value decreases by 1e-5 each time, and the lower limit is reduced to 0.05. In addition, the parameters in each training process will be output and saved, and the model parameters will also be saved every training time.

[0088] For step S4: Finally, the loss function is calculated and the parameters of the network model are updated, where:

[0089] The loss function is obtained by calculating the average square difference between the Q value output by the model and the target Q value according to the following formula.

[0090]

[0091] Then create an Adam optimizer to update the main network parameters to minimize the loss function, and control the speed of model parameter update through the learning rate.

[0092] Furthermore, for step S4, for each new optical path request, the intelligent body will select the optimal path and spectrum allocation plan according to the current network status; then execute the plan, update the network status, and record the parameters of each request including allocation results, network request blocking rate, and spectrum utilization during the training process. By giving appropriate routing path selection and spectrum allocation strategies, the blocking rate in the optical network can be reduced and resource utilization can be improved.

[0093] See also Figure 5 , which is the Input_feature.txt file composed of the input state feature vector of the model. The parameters include the one-hot representation of the source node and the destination node in the lightpath request, the bandwidth size is 97, the index of the candidate path is 0 or 1, the number of continuous spectrum slots that meet the requirements in the link, the number of spectrum slots required to meet the bandwidth requirements, and the maximum and minimum continuous spectrum slot sizes that meet the requirements in the link.

[0094] See also Figure 6 , which is the DQN_feature.txt file composed of data generated during the model training process. The parameters include the total path length from the source node to the destination node, which is 1571, the number of spectrum slots num_FS required to meet the light path request, which is 5, the nodes included in the path from the source node to the destination node [139, 170, 171], the path length span [4, 6, 11], the spectrum starting position, starting from the 0th position and ending at the 4th position, corresponding to the following spectrum resource usage, and the corresponding power spectrum density.

[0095] See also Figure 7 , which is the data generated during the model training process and input into the q_values.xlsx file. The output parameters include ① (source node, destination node pair) in each training process - correspondingly stored in temp; ② the bandwidth size current_bandwidth required by the optical path request; ③ the selected path index path_id (there are two candidate paths, so the index value is 0 or 1); ④ the total length of the selected path path_len; ⑤ the link connection path_links included in the optical path connection; ⑥ the path length lengths between the nodes in the selected path; ⑦ the modulation format modulation_format; ⑧ the number of spectrum num_FS required to meet the optical path request calculated based on the path length and bandwidth requirements; ⑨ the starting position fs_start of the spectrum on the selected link; ⑩ the ending position fs_end of the spectrum on the selected link. If the values ​​of fs_start and fs_end are both -1, it means that the spectrum resources on the selected link do not meet the optical path request, resulting in blocking; The spectrum utilization of the link is resource_util; The Q values ​​q_values ​​of the two candidate paths.

[0096] See also Figure 8, which is a comparison chart of the blocking rate of the traditional algorithm and the algorithm of the present invention. The green line is the traditional heuristic (KSP) algorithm, the blue line is the blocking rate of the traditional DQN algorithm without considering spectrum fragmentation, and the orange line is the blocking rate of the algorithm of the present invention with considering spectrum fragmentation. It can be seen from the image that the routing and spectrum allocation algorithm combined with the DQN algorithm has a significant improvement in the blocking rate compared with the traditional algorithm. Although the blocking rates of the traditional DQN and the algorithm of the present invention are not much different in the early stage, this is because the number of optical path requests is small in the early stage, and the idle spectrum resources in the network link are relatively continuous, so considering spectrum fragmentation does not improve the performance of the algorithm much. However, in the later stage, when the number of optical path requests is large and the idle spectrum resources of the link are relatively scattered, considering the degree of spectrum fragmentation can reduce the blocking rate of requests in the network. In the figure, when the number of optical path requests reaches 3000, the blocking rate of the traditional DQN algorithm without considering spectrum fragmentation is about 0.13, while the algorithm of the present invention is only about 0.1, which can achieve a blocking reduction of about 20%.

[0097] It is worth mentioning that the technical features such as DQN involved in the patent application of this invention should be regarded as the prior art. The specific structure, working principle and possible control method and spatial layout method of these technical features can be selected by conventional methods in the field, and should not be regarded as the inventive point of the patent of this invention. The patent of this invention will not be further elaborated.

[0098] For those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A DQN-based optical network routing and spectrum allocation algorithm, characterized in that: The following steps are involved: Step S1: define the optical network environment and algorithm initialization, and build the basic framework of the DQN model; Step S2: receiving a light path request and executing RSA decision; Step S3: Feedback the corresponding reward according to the executed RSA decision and train the DQN model; Step S4: Test the DQN model and select the optimal path and spectrum allocation scheme for each received lightpath request according to the current network status.

2. According to claim 1, a DQN-based optical network routing and spectrum allocation algorithm is characterized in that: Step S1 is specifically implemented as follows: Step S1.1: Design a DQN model, including an input layer, a hidden layer, and an output layer; the input layer is used to receive information in the state space and convert it into a format for neural network processing; the hidden layer is used to extract state features and learn the mapping relationship between state and action; the output layer is used to output the Q value corresponding to each candidate path; Step S1.2: define the preferred state space in the RSA problem as the input of the DQN model; the preferred state space is the input feature vector composed of features extracted from the network topology connection, spectrum resource usage and lightpath request information, and the parameters in the vector include the One-Hot encoding corresponding to the source node-destination node in the lightpath request, bandwidth, index of the candidate path that meets the demand, the total number of idle spectrum slots, the number of spectrum slots that meet the demand continuously, the maximum size of the idle spectrum slots that meet the demand and the minimum size of the idle spectrum slots that meet the demand; Step S1.3: Define the preferred action space in the RSA problem, that is, select the appropriate path and spectrum allocation scheme; the DQN model outputs the Q value of all candidate paths according to the input feature vector of the state space, selects the path according to the Q value, and then performs spectrum allocation.

3. The DQN-based optical network routing and spectrum allocation algorithm according to claim 2, characterized in that: For step S2, Using the DQN model for RSA decision-making, first extract the features in the state space as the input feature vector of the DQN model. Then the DQN model outputs the Q values ​​of all candidate paths and selects a path based on the Q value before allocating spectrum. Specifically, When allocating spectrum, the appropriate modulation format is first selected according to the transmission distance. After obtaining the number of spectrum slots to be allocated, the spectrum is allocated by first fitting and randomly selecting. For both methods, the formula A = (S + F cut +M align ) / C calculates the spectrum fragmentation degrees caused by the two methods in the current state, and compares them. Finally, the method with the smaller spectrum fragmentation value is adopted to improve the utilization rate of spectrum resources in the network; Where S is the number of spectrum slots required to satisfy the lightpath request, F cut M is the number of links that will be split into two pieces after allocating spectrum at the current location. align is the sum of the number of idle spectrum slots at the same spectrum position in the adjacent links of the allocated link, and C is the number of all continuous idle spectrum slots that meet the spectrum allocation in the current link.

4. The DQN-based optical network routing and spectrum allocation algorithm according to claim 3, characterized in that: For step S3, during the training process, the intelligent body selects actions according to the current state, gets rewards after execution, and stores the experience data in the experience replay buffer. When a certain amount of data is stored in the replay buffer, denoted as m, training will start again. After training, m / 2 earlier generated data in the replay buffer will be deleted to keep the sample data in the replay buffer updated. During the training process, every time the amount of data in the replay buffer is stored to m again, the DQN model will randomly extract a batch of data from the experience replay buffer for training and updating the parameters of the DQN model.

5. The DQN-based optical network routing and spectrum allocation algorithm according to claim 4, characterized in that: For step S4, for each new optical path request, the intelligent body will select the optimal path and spectrum allocation plan according to the current network status; then execute the plan, update the network status, and record the parameters of each request including allocation results, network request blocking rate, and spectrum utilization during the training process. By giving appropriate routing path selection and spectrum allocation strategies, the blocking rate in the optical network is reduced and resource utilization is improved.

Images