Routing and resource allocation method and device in elastic optical network

Abstract

The present application provides a method and apparatus for routing and resource allocation in an elastic optical network, the method comprising: obtaining characteristic information of multiple alternative routes for a target service to be allocated; inputting the characteristic information of each alternative route and the characteristic information of the target service as states into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in an elastic optical network, so that the DDPG reinforcement learning model outputs one of the alternative routes as the working route for the target service in the elastic optical network. The present application can reduce spectrum fragmentation in an elastic optical network, reduce the service blocking rate in an elastic optical network, and effectively improve the transmission capacity of the elastic optical network; by using machine learning methods to optimize an elastic optical network, a better routing and resource allocation strategy can be found, and the reliability of the routing and resource allocation process in the elastic optical network and the effectiveness of the results can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of elastic optical networks, and in particular to a method and apparatus for routing and resource allocation in elastic optical networks. Background Art

[0002] Amidst the rapid development of optical network technology, the emergence of Elastic Optical Networks (EONs) has alleviated the problem of low optical bandwidth resource utilization in optical networks. Compared to traditional fixed-grid (e.g., 50 GHz) wavelength division multiplexing (WDM) network solutions, EON networks can decompose bandwidth resources into smaller granularities and adjust the format based on the transmission quality of the optical path, flexibly establishing variable-bandwidth super-channels. While WDM network technology divides bandwidth resources into channels of equal size, addressing the issue of optical bandwidth waste to a certain extent, the diversification of user services has led to the emergence of services with relatively small bandwidth requirements. These services with relatively small bandwidth requirements also occupy the entire channel wavelength resource during resource allocation, resulting in significant waste of remaining bandwidth resources. EON networks decompose wavelength resources into a finer granularity than WDM networks. EON networks divide available spectrum resources into a set of frequency slots (FSs) with a bandwidth of 6.25 GHz or 12.5 GHz, referred to as spectrum resources. This smaller granularity fully adapts to the needs of dynamic network service development.

[0003] In EON networks, the routing and spectrum assignment (RSA) process for user services must adhere to three major spectrum allocation constraints: spectrum continuity, spectrum consistency, and spectrum non-overlap. Furthermore, each service must retain one or more frequency slots (FSs) to serve as guard bands between services. These issues lead to a large amount of spectrum fragmentation in elastic optical networks, reducing EON network spectrum resource utilization. Consequently, effective optimization of optical networks has garnered widespread attention. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a routing and resource allocation method and apparatus in an elastic optical network to eliminate or improve one or more defects in the prior art.

[0005] One aspect of the present application provides a routing and resource allocation method in an elastic optical network, comprising:

[0006] Obtaining characteristic information of each of multiple candidate routes for the target service to be assigned;

[0007] The characteristic information of each alternative route and the characteristic information of the target service are input as states into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in an elastic optical network, so that the DDPG reinforcement learning model outputs one of the alternative routes as the working route of the target service in the elastic optical network.

[0008] In some embodiments of the present application, further comprising:

[0009] Obtaining characteristic information of each of multiple candidate routes for the service to be allocated;

[0010] Generating an input state of the DDPG reinforcement learning model according to the characteristic information of each candidate route and the characteristic information of the target service;

[0011] Selecting a corresponding execution action based on a preset routing and resource allocation strategy, so that the execution action interacts with the network environment of the elastic optical network, and obtaining a reward for the initial execution action and a next input state of the DDPG reinforcement learning model from the network environment;

[0012] If it is determined that the current iterative process has not ended, the DDPG reinforcement learning model is continued to be trained based on the reward value of the execution action and the next input state until the iterative process ends, thereby obtaining a DDPG reinforcement learning model for optimizing routing and resource allocation in elastic optical networks.

[0013] In some embodiments of the present application, after obtaining the reward for the initial execution action and the next input state of the DDPG reinforcement learning model from the network environment, the method further includes:

[0014] Storing the input state, reward, next input state and completion state identifier as a four-tuple in a preset experience storage data structure;

[0015] Correspondingly, if it is determined that the current iterative process has not ended, before continuing to train the DDPG reinforcement learning model based on the reward value of the executed action and the next input state, the method further includes:

[0016] It is determined whether there is currently any remaining storage space in the experience storage data structure. If so, it is determined that the current iteration process has not ended.

[0017] In some embodiments of the present application, obtaining characteristic information of each of the multiple candidate routes of the target service to be allocated includes:

[0018] receiving routing and resource allocation requests for the elastic optical network;

[0019] Based on the KSP algorithm, multiple candidate routes of the target service corresponding to the route and the resource allocation request are searched, and characteristic information of each candidate route is collected.

[0020] In some embodiments of the present application, inputting the characteristic information of each candidate route and the characteristic information of the target service into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in an elastic optical network includes:

[0021] Generating an input state corresponding to the target service according to the characteristic information of each candidate route and the characteristic information of the target service, wherein the characteristic information of the target service includes: a start node, an end node, and a required bandwidth of the target service;

[0022] The input state is input into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in an elastic optical network.

[0023] In some embodiments of the present application, further comprising:

[0024] If a working route is successfully assigned to the target service, a corresponding reward is returned based on a preset spectrum fragmentation degree metric;

[0025] If the work route is not successfully assigned to the target business, a fixed reward value of -1 is returned.

[0026] In some embodiments of the present application, the Actor network model and the Critic network model corresponding to the DDPG reinforcement learning model both adopt a 4-layer deep neural network DNN;

[0027] The activation functions of the Actor network model and the Critic network model both adopt the ReLU function; and the optimization functions of the Actor network model and the Critic network model both adopt the Adam function.

[0028] Another aspect of the present application provides a routing and resource allocation device in an elastic optical network, comprising:

[0029] The alternative route acquisition module is used to obtain characteristic information of multiple alternative routes of the target service to be allocated;

[0030] A DDPG agent is used to input the characteristic information of each alternative route and the characteristic information of the target service as state into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in an elastic optical network, so that the DDPG reinforcement learning model outputs one of the alternative routes as the working route of the target service in the elastic optical network.

[0031] On the other hand, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the routing and resource allocation method in the elastic optical network when executing the computer program.

[0032] On the other hand, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the routing and resource allocation method in the elastic optical network when executed by a processor.

[0033] The routing and resource allocation method in an elastic optical network provided in this application obtains characteristic information of multiple candidate routes for a target service to be allocated; the characteristic information of each candidate route and the characteristic information of the target service are input as states into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in the elastic optical network, so that the DDPG reinforcement learning model outputs one of the candidate routes as the working route for the target service in the elastic optical network. Specifically, firstly, this application utilizes a relatively advanced DDPG reinforcement learning algorithm to optimize the elastic optical network, reducing spectrum fragmentation in the network. Simultaneously, it also reduces the service blocking rate in the network and improves the network's transmission capacity. Secondly, by using machine learning methods to optimize the optical network, better routing and resource allocation strategies can be found. Furthermore, the DDPG reinforcement learning algorithm converges more easily than the actor-critic algorithm and addresses the correlation between training data, thereby improving the reliability of the routing and resource allocation process in the elastic optical network and the effectiveness of the results.

[0034] Additional advantages, purposes, and features of the present application will be described in part in the following description and will become apparent to those skilled in the art upon study of the following or may be learned from practice of the present application. The purposes and other advantages of the present application may be achieved and obtained by the structures specifically pointed out in the specification and drawings.

[0035] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are intended to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger than other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0037] Figure 1 Schematic diagram of the overall flow of a routing and resource allocation method in an elastic optical network in one embodiment of the present application.

[0038] Figure 2 The figure is a schematic diagram of a specific flow chart of a routing and resource allocation method in an elastic optical network in one embodiment of the present application.

[0039] Figure 3 FIG. 1 is a structural diagram of a routing and resource allocation device in an elastic optical network in another embodiment of the present invention.

[0040] Figure 4 A schematic diagram of state s provided for the application example of this application.

[0041] Figure 5 Schematic diagram of spectrum usage for alternative routes provided for this application example.

[0042] Figure 6 Schematic diagram of the ReLU function provided for the application example of this application.

[0043] Figure 7 Schematic diagram of the Tanh function provided for the application example of this application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail in conjunction with the embodiments and drawings. Here, the illustrative embodiments of this application and their descriptions are used to explain this application, but are not intended to limit this application.

[0045] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show structures and / or processing steps that are closely related to the scheme according to the present application, while other details that are not closely related to the present application are omitted.

[0046] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0047] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0048] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0049] The EON network is considered to be an effective solution to various problems in the next generation of optical networks. In order to effectively solve the dynamic RSA problem in the EON network, the applicant proposed an efficient multi-constraint routing algorithm, the sorted feasible path search algorithm SFPS, in an existing technology, and decomposed the problem into RSA sub-problems. In the context of dynamic traffic, the shortest feasible path is found, and the completeness, optimality and complexity of the SFPS strategy they proposed are proved. At the same time, in terms of spectrum allocation, two allocation strategies, fixed splitting and adaptive splitting, are also proposed. Simulation results show that the proposed dynamic RSA strategy has relatively good time performance, and in general scenarios, it is very good in terms of bandwidth blocking probability and spectrum fragmentation rate.

[0050] In another existing approach, researchers have focused on achieving routing and resource allocation performance while reducing computational complexity. First, a dynamic network resource assessment method is proposed that simultaneously considers the distribution of traffic bandwidth and the carrying capacity of spectrum blocks. Based on this, the concept of traffic-based fragmentation awareness is introduced into the RSA step. In the routing process, a low-complexity shard-aware load-balanced shortest path routing scheme and an improved shard-aware load-balanced k-shortest path routing scheme are proposed. In the spectrum allocation step, an effective traffic-based segment-aware spectrum allocation scheme is proposed.

[0051] There is also a fragmentation-aware VONE algorithm in the existing technology to reduce spectrum fragmentation and improve resource unitization. First, an integer linear programming model ILP is formulated for multiple static service requests, while considering the continuity and continuity of the substrate fiber link SFL spectrum. Then, in order to realize the coordination node and link mapping in VONE, a virtual auxiliary graph VAG is constructed, which decomposes the physical infrastructure into multiple hierarchical graphs according to the spectrum time slot requirements of the VONE request. The first two benchmark schemes (i.e., spectrum resource traffic balancing and spectrum resource and computing resource traffic balancing) apply traffic balancing strategies to spectrum resources and computing resources, while the other spectrum resource traffic balancing and defragmentation SRTB&DF scheme additionally uses the defragmentation method in the active VONE process. At the same time, this approach also takes into account static network planning and dynamic service provision scenarios.

[0052] However, the above method cannot reduce the spectrum fragmentation rate in the elastic optical network while reducing the service blocking rate and performance consumption in the elastic optical network. Therefore, in order to solve this problem, the present application improves a routing and resource allocation method in the elastic optical network based on machine learning, which is an elastic optical network optimization method.

[0053] First, this application uses the advanced DDPG reinforcement learning algorithm to optimize elastic optical networks and reduce spectrum fragmentation. This also reduces service congestion and improves network transmission capacity. Second, using machine learning to optimize optical networks can help find better routing and resource allocation strategies.

[0054] The DDPG reinforcement learning algorithm also converges more easily than the Actor-Critic algorithm and addresses the correlation between training data. It is also more performance-efficient than the A3C algorithm, which uses a multi-threaded asynchronous learning method to learn policies, which inevitably increases performance overhead.

[0055] Based on this, the embodiment of the present application provides a routing and resource allocation method in an elastic optical network, see Figure 1 The routing and resource allocation method in the elastic optical network specifically includes the following contents:

[0056] Step 100: Obtain characteristic information of each of multiple candidate routes of the target service to be allocated.

[0057] In step 100, k candidate routes are first found, and key feature information of these k candidate routes is collected, for example, the occupancy of spectrum resources on each path.

[0058] Step 200: Input the characteristic information of each alternative route and the characteristic information of the target service as states into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in an elastic optical network, so that the DDPG reinforcement learning model outputs one of the alternative routes as the working route of the target service in the elastic optical network.

[0059] In one or more embodiments of the present application, the DDPG reinforcement learning model can also be written as a DDPG reinforcement learning network model.

[0060] Among them, the embodiments of the present application can also adopt other types of reinforcement learning algorithms, such as Q-learning, DQL, DDQN, Actor-Critic, A3C, etc.

[0061] As you can see, the agent selects action A based on the current input state S and follows a specific strategy. This action interacts with the EON network environment. The agent receives the reward R for action A and the next input state S' from the EON network environment. After a certain number of training rounds, the DDPG agent interacts extensively with the EON network environment, enabling it to select the optimal RSA allocation strategy.

[0062] From the above description, it can be seen that the routing and resource allocation method in the elastic optical network provided in the embodiment of the present application can reduce spectrum fragmentation in the elastic optical network, reduce the service blocking rate in the elastic optical network, and effectively improve the transmission capacity of the elastic optical network; and the use of machine learning methods to optimize the elastic optical network can find better routing and resource allocation strategies, thereby improving the reliability of the routing and resource allocation process in the elastic optical network and the effectiveness of the results.

[0063] In order to further improve the application reliability of the DDPG reinforcement learning model, in the routing and resource allocation method in the elastic optical network provided in the embodiment of the present application, see Figure 2 The routing and resource allocation method in the elastic optical network may further include the following contents before step 100:

[0064] Step 010: Obtain characteristic information of each of the multiple candidate routes for the service to be allocated;

[0065] Step 020: Generate the input state of the DDPG reinforcement learning model according to the characteristic information of each candidate route and the characteristic information of the target service;

[0066] Step 030: Selecting a corresponding execution action based on a preset routing and resource allocation strategy, so that the execution action interacts with the network environment of the elastic optical network, and obtaining a reward for the initial execution action and a next input state of the DDPG reinforcement learning model from the network environment;

[0067] Among them, the selection of the corresponding execution action based on the preset routing and resource allocation strategy can specifically be: after inputting the DDPG reinforcement learning model based on the above state, the corresponding execution action is obtained.

[0068] Step 040: If it is determined that the current iterative process has not ended, the DDPG reinforcement learning model is continued to be trained based on the reward value of the execution action and the next input state until the iterative process ends, thereby obtaining a DDPG reinforcement learning model for optimizing routing and resource allocation in elastic optical networks.

[0069] In one example, the process of training the DDPG reinforcement learning model from step 010 to step 040 can be as shown in Table 1, that is, the DDPG-SF-RSA strategy.

[0070] Table 1 DDPG-SF-RSA strategy

[0071]

[0072]

[0073] In order to further improve the reliability of the storage method of the experience storage part data, in the routing and resource allocation method in the elastic optical network provided in the embodiment of the present application, see Figure 2 The routing and resource allocation method in the elastic optical network further includes the following between step 020 and step 030:

[0074] Step 021: The input state, reward, next input state and completion state identifier are stored as a four-tuple in a preset experience storage data structure.

[0075] For corresponding reference, see Figure 2 The routing and resource allocation method in the elastic optical network further includes the following between step 030 and step 040:

[0076] Step 031: Determine whether there is currently any remaining storage space in the experience storage data structure. If so, determine that the current iteration process has not ended.

[0077] Among them, if it is determined that there is no remaining storage space in the experience storage data structure, the data in the storage space is used to update the network model parameters. After the update is completed, the data in the storage space is cleared.

[0078] Specifically, in the Elastic Optical Network (EON), routing and resource allocation are subject to three major constraints, including the randomness of the start and end nodes of user-demand services and the network spectrum resource allocation restrictions. These issues lead to a large amount of spectrum fragmentation within the network's link resources. Therefore, based on the DDPG strategy algorithm, the storage method for the empirical data storage has been modified.

[0079] In order to further improve the reliability of selecting multiple alternative routes, in a routing and resource allocation method in an elastic optical network provided in an embodiment of the present application, see Figure 2 Step 100 of the routing and resource allocation method in the elastic optical network specifically includes the following:

[0080] Step 110: Receive a routing and resource allocation request for the elastic optical network.

[0081] Step 120: searching for multiple candidate routes for the target service corresponding to the route and resource allocation request based on the KSP algorithm, and collecting characteristic information of each candidate route.

[0082] Specifically, the KSP (k-shortest paths) algorithm is used to find k candidate routes based on the services to be assigned, and key feature information on these k candidate routes is collected, such as the spectrum resource occupancy on each path.

[0083] In order to further optimize the state design, in the routing and resource allocation method in the elastic optical network provided in the embodiment of the present application, see Figure 2 Step 200 of the routing and resource allocation method in the elastic optical network specifically includes the following:

[0084] Step 210: Generate an input state corresponding to the target service based on the characteristic information of each candidate route and the characteristic information of the target service, wherein the characteristic information of the target service includes: the start node, end node and service required bandwidth of the target service;

[0085] Step 220: Input the input state into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in an elastic optical network.

[0086] Specifically, in the design of the state, the application directly uses the first matching FF algorithm on a certain path to calculate the state information in spectrum resource allocation. In this application, a one-dimensional array of length 1×(2|V|+5K) is used as the state s(state), which contains the service request r and k alternative paths for the service. The state s(state) is defined as shown in formula (1):

[0087]

[0088] In this state s, this application uses a one-dimensional array of length 2|V| to represent the characteristics of the start node and the end node, and adopts a one-hot format, which is also o and d in the expression of state s in formula (1). They represent the start node and the end node of the service respectively. k represents the alternative path numbered k among the K alternative paths. For the alternative path numbered k, It represents the size of the first spectrum block that can be used. Indicates the position index of the first spectrum block that can be used, It represents the number of spectrum blocks of the service. Indicates the average size of spectrum blocks that can be used on the route. Indicates the total number of spectrum blocks that can be used.

[0089] When using the KSP algorithm to calculate K alternative routes, if the number of alternative paths from the start node o to the end node d of a certain service is less than K, we set the paths that cannot meet the conditions to a one-dimensional array of -1 to ensure that the format of each service state s is consistent with the format of other normal states s.

[0090] Considering the spectrum fragmentation problem in the EON network, in an embodiment of the present application, a routing and resource allocation method in an elastic optical network provided by step 200 further specifically includes the following content:

[0091] Step 300: If the target service is successfully assigned a working route, a corresponding reward is returned based on a preset spectrum fragmentation degree metric. Step 400: If the target service is not successfully assigned a working route, a fixed reward value of -1 is returned.

[0092] Since step 200 can be performed by a DDPG agent, in step 300, if the action performed by the DDPG agent successfully assigns a working route to the target service, a corresponding reward is returned based on a preset spectrum fragmentation metric. Similarly, in step 400, if the action performed by the DDPG agent fails to successfully assign a working route to the target service, a fixed reward value of -1 is returned.

[0093] Specifically, the DDPG-SF-RSA method takes into account the spectrum fragmentation problem in the EON network. Therefore, in the design of the reward value, the EON network spectrum fragmentation measurement value is added to reflect the impact of EON network spectrum fragmentation on the reward. In the design of the reward return value, if the user service is successfully allocated, the reward value returned is 1+3×(1-FX xt ), if the current business cannot be allocated, the reward value -1 will be returned directly. The reward return value of this strategy is as follows

[0094] As shown in formula 2:

[0095]

[0096] In Formula 2, F ext The meaning of is the spectrum fragmentation degree metric value of the route corresponding to the action. For convenience, the following formula is used here to describe the spectrum fragmentation degree on the current link:

[0097]

[0098] In formula (3), FS max Used to describe the maximum number of consecutive idle spectrum resources in the link spectrum resources, N fre It represents the sum of all unused spectrum resources on the link. ext The closer it is to 1, the more dispersed the unused spectrum resource blocks are, and the higher the degree of spectrum resource fragmentation, the less efficient the resource utilization is. extIf it is close to 0, it means that the current degree of spectrum resource fragmentation is lower, the strategy for allocating spectrum resources is excellent, and the utilization rate of spectrum resources is high.

[0099] In order to further improve the reliability and effectiveness of the DDPG reinforcement learning model, in a routing and resource allocation method in an elastic optical network provided in an embodiment of the present application, the Actor network model and the Critic network model corresponding to the DDPG reinforcement learning model in the routing and resource allocation method in the elastic optical network both adopt a 4-layer deep neural network DNN;

[0100] The activation functions of the Actor network model and the Critic network model both adopt the ReLU function; and the optimization functions of the Actor network model and the Critic network model both adopt the Adam function.

[0101] It is understandable that the DDPG-SF-RSA strategy uses a 4-layer deep neural network (DNN) in its network model design. The ReLU function is activated by the ReLU function. The characteristic of the ReLU function is that when the input x is greater than 0, the output value is x, and when it is less than 0, the output value is 0. The expression of the ReLU function is shown in Formula 4:

[0102]

[0103] The advantage of the ReLU activation function is that it makes it easier for the network parameters of the DDPG agent to converge during the interaction between the DDPG agent and the environment. In addition, during the interaction process, it can make the DDPG agent train faster. At the same time, it also increases the nonlinearity of the network, prevents gradient disappearance, makes the grid sparse, and reduces overfitting. In terms of the output of the Actor network model, the Tanh function is selected. The characteristic of the Tanh function is that no matter whether the input value x is large or small, the output value range will be determined between -1 and 1. The expression of the Tanh function is as shown in formula (5):

[0104]

[0105] The Critic network model also uses the ReLU function as the activation function, and a four-layer deep neural network (DNN) is employed. The Adam optimization function is used for both the Actor and Critic network models. The Adam optimization function uses the first-order and second-order moments of the gradient to dynamically adjust the learning rate of the network model parameters. Essentially, it is RMSprop with a momentum term.

[0106] From the software level, the present application also provides an elastic optical network routing and resource allocation device for executing all or part of the routing and resource allocation method in the elastic optical network, see Figure 3 The routing and resource allocation device in the elastic optical network specifically includes the following contents:

[0107] The alternative route acquisition module 10 is used to obtain characteristic information of multiple alternative routes of the target service to be allocated;

[0108] The DDPG agent 20 is used to input the characteristic information of each alternative route and the characteristic information of the target service as state into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in the elastic optical network, so that the DDPG reinforcement learning model outputs one of the alternative routes as the working route of the target service in the elastic optical network.

[0109] Among them, the DDPG agent can be written as DDPG agent agent.

[0110] The embodiment of the routing and resource allocation device in the elastic optical network provided in this application can be specifically used to execute the processing flow of the embodiment of the routing and resource allocation method in the elastic optical network in the above-mentioned embodiment. Its functions are not repeated here, and reference can be made to the detailed description of the embodiment of the routing and resource allocation method in the above-mentioned elastic optical network.

[0111] The portion of the routing and resource allocation performed by the elastic optical network routing and resource allocation apparatus can be executed on a server. In another practical application scenario, all operations can also be completed on the client device. The specific selection can be based on the processing capabilities of the client device and the limitations of the user's usage scenario. This application is not limited to this. If all operations are completed on the client device, the client device may also include a processor for specific processing of routing and resource allocation in the elastic optical network.

[0112] The client device may include a communication module (i.e., a communication unit) that can establish a communication connection with a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0113] The server and the client device may communicate using any suitable network protocol, including network protocols that have not yet been developed as of the filing date of this application. Examples of such network protocols include TCP / IP, UDP / IP, HTTP, and HTTPS. Furthermore, examples of such network protocols include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols, which are used on top of the aforementioned protocols.

[0114] From the above description, it can be seen that the routing and resource allocation device in the elastic optical network provided in the embodiment of the present application can reduce spectrum fragmentation in the elastic optical network, reduce the service blocking rate in the elastic optical network, and effectively improve the transmission capacity of the elastic optical network; and the use of machine learning methods to optimize the elastic optical network can find better routing and resource allocation strategies, thereby improving the reliability of the routing and resource allocation process in the elastic optical network and the effectiveness of the results.

[0115] To further illustrate this solution, the present application also provides a specific application example of a routing and resource allocation method in an elastic optical network. The routing and resource allocation method in an elastic optical network includes the following content:

[0116] Basic Principle: In the Elastic Optical Network (EON), routing and resource allocation are subject to the randomness of user service start and end nodes, as well as three major constraints on spectrum resource allocation within the service allocation network. These issues lead to a large amount of spectrum fragmentation within the network's link resources. This method, based on the DDPG strategy algorithm, modifies the storage method for the empirical data. This method considers spectrum fragmentation in EON networks and, based on the reinforcement learning algorithm of Deep Deterministic Policy Gradient (DDPG), proposes DDPG-SF-RSA (DDPG-Spectrum Fragmentation-RSA) to optimize routing and resource allocation in EON networks.

[0117] The main principles of the DPG-SF-RSA method are:

[0118] 1) First, the K-shortest paths (KSP) algorithm is used to find k candidate routes based on the service to be assigned, and key feature information on these k candidate routes is collected, such as the spectrum resource occupancy on each path.

[0119] 2) Then, the DDPG agent uses the characteristic information of these k alternative routes and the characteristic information of the user service (demand), such as the starting node, ending node, and service demand bandwidth, as the input state S of the DDPG agent.

[0120] 3) The agent then selects action A based on the current input state S and follows a specific strategy. This action A interacts with the EON network environment. The agent receives the reward R for action A and the next input state S′ from the EON network environment. After a certain number of training rounds, the DDPG agent interacts extensively with the EON network environment, enabling it to select the optimal RSA allocation strategy.

[0121] The pseudo code of the algorithm process is shown in Table 1.

[0122] This method uses DDPG reinforcement learning, so here we introduce the model design of this method, focusing on the design process of state representation, agent action space, and reward.

[0123] (1) State design

[0124] In the design of the state, the application directly uses the first matching FF algorithm on a certain path to calculate the state information in spectrum resource allocation. In this application, a one-dimensional array of length 1×(2|V|+5K) is used as the state s(state), which contains the service request r and the k alternative paths for the service. The state s(state) is defined as shown in formula (1):

[0125]

[0126] In this state s, this application uses a one-dimensional array of length 2|V| to represent the characteristics of the start node and the end node, and adopts a one-hot format, which is also o and d in the expression of state s in formula (1). They represent the start node and the end node of the service respectively. k represents the alternative path numbered k among the K alternative paths. For the alternative path numbered k, It represents the size of the first spectrum block that can be used. Indicates the position index of the first spectrum block that can be used, It represents the number of spectrum blocks of the service. Indicates the average size of spectrum blocks that can be used on the route. Indicates the total number of spectrum blocks that can be used.

[0127] When using the KSP algorithm to calculate K alternative routes, if the number of alternative paths from the start node o to the end node d of a certain service is less than K, we set the paths that cannot meet the conditions to a one-dimensional array of -1 to ensure that the format of each service state s is consistent with the format of other normal states s.

[0128] exist Figure 4 In the figure, the state structure when K=2 is described. Business D is in Figure 4 In the figure, for a service from 1 node to 4 nodes, the number of frequency slots (FSs) required is 2. Figure 4 The right side shows the resource usage of the two alternative routes of business D and the structure of state s.

[0129] (2) Action Space

[0130] The DDPG-SF-RSA method selects one working route from K candidate paths for each service request. Therefore, the action space of this strategy is consistent with the parameter K in the KSP algorithm used in this strategy. In this action space, this method contains K actions.

[0131] (3) Rewards

[0132] The DDPG-SF-RSA method takes into account the spectrum fragmentation problem in the EON network. Therefore, in the design of the reward value, the EON network spectrum fragmentation measurement value is added to reflect the impact of EON network spectrum fragmentation on the reward. In the design of the reward return value, if the user service is successfully allocated, the reward value returned is 1+3×(1-F ext ), if the current business cannot be allocated, the reward value -1 is directly returned. The reward return value of this strategy is shown in Formula 2:

[0133]

[0134] In formula (2), F ext The meaning of is the spectrum fragmentation degree metric value of the route corresponding to the action. For convenience, the following formula is used here to describe the spectrum fragmentation degree on the current link:

[0135]

[0136] In formula (3), FS max Used to describe the maximum number of consecutive idle spectrum resources in the link spectrum resources, N fre It represents the sum of all unused spectrum resources on the link. extThe closer it is to 1, the more dispersed the unused spectrum resource blocks are, and the higher the degree of spectrum resource fragmentation, the less efficient the resource utilization is. ext If it is close to 0, it means that the current degree of fragmentation of spectrum resources is lower, the strategy for allocating spectrum resources is excellent, and the utilization rate of spectrum resources is high. This application does not limit this.

[0137] It is understandable that there are many formulas for evaluating the degree of spectrum fragmentation in a network. For example, formula (3) used in this method is only an example, and other formulas for describing spectrum fragmentation on a link may also be used.

[0138] exist Figure 5 In the figure, the spectrum resource usage of the alternative routes for a certain service is shown. Assume that there are 13 frequency slots (FSs) on each link in the EON network, and the spectrum resource usage of the alternative links for the current service is given. If the DDPG-SF-RSA method selects the following paths respectively, the reward values ​​are R path1 =1+3*7 / 8=3.625, R path2 =1+3*2 / 6=2, R path3 =1+3*3 / 8=2.215, R path4 =1+3*3 / 7=2.29, R path5 =1+3*3 / 8=2.215. As you can see, different strategies give different actions, and the returned reward values ​​are also different.

[0139] (4) Network model

[0140] In the network model design of the DDPG-SF-RSA strategy, the Actor network model uses a 4-layer deep neural network DNN, and the activation function uses the ReLU function. The characteristic of the ReLU function is that when the input x is greater than 0, the output value is x, and when it is less than 0, the output value is 0. The image of the ReLU function is as follows Figure 6 , the expression of the ReLU function is shown in formula (4):

[0141]

[0142] The advantage of the ReLU activation function is that it makes it easier for the network parameters of the DDPG agent to converge during the interaction between the DDPG agent and the environment. In addition, during the interaction, it can make the DDPG agent train faster, while also increasing the nonlinearity of the network, preventing the gradient from disappearing, making the grid sparse, and reducing overfitting. In terms of the output of the Actor network model, the Tanh function is selected. The characteristic of the Tanh function is that no matter whether the input value x is large or small, the output value range will be determined between -1 and 1. The expression of the Tanh function is as shown in formula (5), and the image is as shown in Figure 7 As shown:

[0143]

[0144] The Critic network model also uses the ReLU function as the activation function, and a four-layer deep neural network (DNN) is employed. The Adam optimization function is used for both the Actor and Critic network models. The Adam optimization function uses the first-order and second-order moments of the gradient to dynamically adjust the learning rate of the network model parameters. Essentially, it is RMSprop with a momentum term.

[0145] Here, the main advantages of the Adam optimization function are:

[0146] (a) Efficient computation and uses less memory;

[0147] (b) Suitable for solving problems with relatively large data scale;

[0148] (c) The Adam optimization function combines the advantages of both the AdaGrad and RMSprop algorithms and is very suitable for solving problems with high noise.

[0149] (d) Hyperparameters are easy to observe intuitively and do not require a large number of tuning parameters.

[0150] (5) EON network environment

[0151] The main function of the EON network environment is to interact with the intelligent agent of the DDPG-SF-RSA method to simulate the real EON network routing and resource allocation process. In terms of actual design, this application uses Python code simulation to implement the simulation environment of the EON network. This network environment mainly implements the network environment of single-core and multi-core optical fibers. It also contains the function of simulating network topology, and can modify some RSA algorithms to perform routing and spectrum resource allocation for specific services. At the same time, this network environment returns different reward values ​​according to different actions taken, and the reward calculation is as shown in formula (2). At the same time, this network environment service also randomly generates user services (demand) of requests.

[0152] To sum up, the application example of this application proposes a routing and resource allocation method for elastic optical networks. First, the DDPG algorithm used is one of the most popular reinforcement learning algorithms. The biggest advantage of this algorithm is that it can converge quickly and find solutions to problems. At the same time, DDPG uses a dual network model and experience recycling mechanism, which to a certain extent solves the problem of the actor-critic algorithm being difficult to converge, and also solves the correlation problem between training, so that the problem to be solved can better obtain the corresponding strategy.

[0153] Secondly, by using reinforcement learning to interact with the environment, the resulting user service routing and resource allocation strategies will undoubtedly surpass human-generated strategies. This can effectively optimize spectrum fragmentation in optical networks, preventing the generation of large amounts of spectrum fragmentation and increasing the transmission capacity of optical networks.

[0154] 1. The optimization is to solve the spectrum fragmentation problem in the EON network.

[0155] 2. It adopts the more advanced DDPG reinforcement learning algorithm.

[0156] 3. The storage part of the empirical data in the DDPG algorithm is customized to facilitate training and obtain the desired solution.

[0157] To address spectrum fragmentation in single-band (C-band) EON networks, this application proposes a strategy for optimizing routing and resource allocation for spectrum fragmentation in the network based on the reinforcement learning DDPG algorithm. This strategy aims to reduce spectrum fragmentation in single-band EON networks, increase network transmission capacity, and enhance overall network performance.

[0158] The embodiment of the present application also provides a computer device (i.e., an electronic device), which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the routing and resource allocation method in the elastic optical network mentioned in the above embodiment, wherein the processor and the memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and the memory via a wired or wireless manner. The computer device is communicatively connected to the routing and resource allocation device in the elastic optical network to receive real-time motion data from the sensors in the wireless multimedia sensor network and receive original video sequences from the video acquisition device.

[0159] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0160] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the routing and resource allocation method in an elastic optical network described in the embodiments of this application. The processor executes the non-transitory software programs, instructions, and modules stored in the memory to perform various processor functions and data processing, thereby implementing the routing and resource allocation method in an elastic optical network described in the aforementioned method embodiments.

[0161] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The one or more modules are stored in the memory, and when executed by the processor, perform the routing and resource allocation method in the elastic optical network in the embodiment.

[0163] In some embodiments of the present application, the user equipment may include a processor, a memory and a transceiver unit, and the transceiver unit may include a receiver and a transmitter. The processor, memory, receiver and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0164] As an implementation method, the functions of the receiver and transmitter in this application can be considered to be implemented through a transceiver circuit or a dedicated transceiver chip, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general-purpose chip.

[0165] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiments of the present application. That is, the program code for implementing the functions of the processor, receiver, and transmitter is stored in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0166] The present application also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the steps of the aforementioned method for routing and resource allocation in an elastic optical network. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0167] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0168] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0169] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0170] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Those skilled in the art will appreciate that various modifications and variations of the present embodiment are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A routing and resource allocation method in an elastic optical network, characterized in that: include: Obtaining characteristic information of each of multiple candidate routes for the service to be allocated; Generate the input state of the DDPG reinforcement learning model based on the characteristic information of each alternative route and the characteristic information of the target service; Selecting a corresponding execution action based on a preset routing and resource allocation strategy, so that the execution action interacts with the network environment of the elastic optical network, and obtaining a reward for the initial execution action and a next input state of the DDPG reinforcement learning model from the network environment; If it is determined that the current iterative process has not ended, then continuing to train the DDPG reinforcement learning model based on the reward value of the executed action and the next input state until the iterative process ends, thereby obtaining a DDPG reinforcement learning model for optimizing routing and resource allocation in the elastic optical network; Obtaining characteristic information of each of multiple candidate routes for the target service to be assigned; Inputting characteristic information of each candidate route and characteristic information of the target service as states into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in the elastic optical network, so that the DDPG reinforcement learning model outputs one of the candidate routes as a working route for the target service in the elastic optical network; The step of obtaining characteristic information of each of the plurality of candidate routes for the target service to be allocated includes: receiving routing and resource allocation requests for the elastic optical network; Based on the KSP algorithm, multiple candidate routes of the target service corresponding to the route and the resource allocation request are searched, and characteristic information of each candidate route is collected.

2. The routing and resource allocation method in an elastic optical network according to claim 1, characterized in that: After obtaining the reward for the initial execution action and the next input state of the DDPG reinforcement learning model from the network environment, the method further includes: Storing the input state, reward, next input state and completion state identifier as a four-tuple in a preset experience storage data structure; Correspondingly, if it is determined that the current iterative process has not ended, before continuing to train the DDPG reinforcement learning model based on the reward value of the executed action and the next input state, the method further includes: It is determined whether there is currently any remaining storage space in the experience storage data structure. If so, it is determined that the current iteration process has not ended.

3. The routing and resource allocation method in an elastic optical network according to claim 1, characterized in that: The inputting the characteristic information of each candidate route and the characteristic information of the target service into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in the elastic optical network includes: Generating an input state corresponding to the target service according to the characteristic information of each candidate route and the characteristic information of the target service, wherein the characteristic information of the target service includes: a start node, an end node, and a required bandwidth of the target service; The input state is input into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in an elastic optical network.

4. The routing and resource allocation method in an elastic optical network according to claim 1, characterized in that: Also includes: If a working route is successfully assigned to the target service, a corresponding reward is returned based on a preset spectrum fragmentation degree metric; If the work route is not successfully assigned to the target business, a fixed reward value of -1 is returned.

5. The routing and resource allocation method in an elastic optical network according to any one of claims 1 to 4, characterized in that: The Actor network model and Critic network model corresponding to the DDPG reinforcement learning model both use a 4-layer deep neural network DNN; The activation functions of the Actor network model and the Critic network model both adopt the ReLU function; and the optimization functions of the Actor network model and the Critic network model both adopt the Adam function.

6. A routing and resource allocation device in an elastic optical network, characterized in that: The routing and resource allocation device in the elastic optical network is further configured to perform the following: Obtaining characteristic information of each of multiple candidate routes for the service to be allocated; Generate the input state of the DDPG reinforcement learning model based on the characteristic information of each alternative route and the characteristic information of the target service; Selecting a corresponding execution action based on a preset routing and resource allocation strategy, so that the execution action interacts with the network environment of the elastic optical network, and obtaining a reward for the initial execution action and a next input state of the DDPG reinforcement learning model from the network environment; If it is determined that the current iterative process has not ended, then continuing to train the DDPG reinforcement learning model based on the reward value of the executed action and the next input state until the iterative process ends, thereby obtaining a DDPG reinforcement learning model for optimizing routing and resource allocation in the elastic optical network; The routing and resource allocation device in the elastic optical network also includes: The alternative route acquisition module is used to obtain characteristic information of multiple alternative routes of the target service to be allocated; a DDPG agent, configured to input characteristic information of each candidate route and characteristic information of the target service as states into a preset DDPG reinforcement learning model for optimizing routing and resource allocation in the elastic optical network, so that the DDPG reinforcement learning model outputs one of the candidate routes as a working route for the target service in the elastic optical network; The step of obtaining characteristic information of each of the plurality of candidate routes for the target service to be allocated includes: receiving routing and resource allocation requests for the elastic optical network; Based on the KSP algorithm, multiple candidate routes of the target service corresponding to the route and the resource allocation request are searched, and characteristic information of each candidate route is collected.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the routing and resource allocation method in an elastic optical network according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the routing and resource allocation method in an elastic optical network according to any one of claims 1 to 5 is implemented.

Images