Reinforcement learning-based all-optical network topology reconfiguration method and system

Abstract

The present application relates to the technical field of optical switching, and in particular to a full-optical-network topology reconfiguration method and system based on reinforcement learning, which solves the technical problem of insufficient reconstruction flexibility and service scenario adaptability in the topology reconfiguration process caused by one-sided state representation, flat action space and lagging reward feedback in the prior art. The method comprises: constructing a full-optical-network physical topology structure; obtaining network operation state information of the full-optical network, and constructing a multi-dimensional state vector; according to the full-optical-network physical topology structure, the network operation state information and the multi-dimensional state vector, layer-by-layer decision is made in a hierarchical action space to generate a topology reconfiguration strategy; and according to the topology reconfiguration strategy, cross-connection instructions are generated and delivered to a reconfigurable optical add-drop multiplexer for execution to complete topology reconfiguration of the full-optical network.

Description

Technical Field

[0001] This invention relates to the field of optical switching technology, and specifically to a method and system for topology reconstruction of all-optical networks based on reinforcement learning. Background Technology

[0002] As the core architecture of next-generation optical communication networks, all-optical networks enable direct switching and scheduling of optical signals at the physical layer through reconfigurable optical add-drop multiplexers (ROADMs), facilitating flexible service add-drop and cross-connection without optoelectronic conversion. In service scenarios with significant traffic fluctuations, all-optical network topology reconfiguration technology can dynamically adjust wavelength resource allocation strategies based on real-time service demands, performing wavelength pass-through, addition, and drop operations. This ensures sufficient bandwidth supply during peak periods and reduces overall network energy consumption during off-peak periods, making it a key means to improve the resource utilization efficiency and adaptability of all-optical networks.

[0003] In recent years, reinforcement learning algorithms have been increasingly adopted in the field of all-optical network topology reconstruction due to their superior performance in sequential decision-making problems. Deep Q-networks, through experience replay mechanisms and collaborative training with the target network, can effectively fit state-action value functions in complex network environments, allocating transmission paths and wavelength resources for dynamic connection requests. Existing all-optical network topology reconstruction methods based on deep Q-networks typically treat route selection and wavelength allocation as unified decision objectives, guiding a pre-designed deep reinforcement learning network to learn the optimal strategy through a reward function, aiming to achieve a balance between network congestion rate, wavelength utilization, and transmission loss.

[0004] However, the resource requirements of nodes in an all-optical network change dynamically with service traffic, and the physical characteristics of optical layer transmission also fluctuate in real time. This makes it difficult for the reward function in the traditional reinforcement learning framework to accurately reflect the real-time changes in network state. At the same time, the traditional flat action space cannot adapt to the multi-dimensional control capabilities of reconfigurable optical add-drop multiplexers in terms of path, wavelength, and port, resulting in an adaptation deviation between topology reconstruction decisions and actual service scenarios. The reconfigured network topology is difficult to accurately match the dynamically changing resource requirements and physical transmission constraints. Summary of the Invention

[0005] To address the technical problems of insufficient flexibility and adaptability to business scenarios in topology reconstruction due to incomplete state representation, flat action space, and delayed reward feedback in existing technologies, this invention aims to provide a method and system for topology reconstruction of all-optical networks based on reinforcement learning. The specific technical solution adopted is as follows: Firstly, a reinforcement learning-based all-optical network topology reconfiguration method is provided, comprising: acquiring the connection relationships between reconfigurable optical add-drop multiplexers and the link lengths of optical fiber links in the all-optical network to construct the physical topology of the all-optical network; acquiring the network operation status information of the all-optical network to construct a multi-dimensional state vector; making decisions layer by layer in the hierarchical action space based on the physical topology of the all-optical network, the network operation status information, and the multi-dimensional state vector to determine the path optimization action, wavelength scheduling action, and link switching action corresponding to the optimal transmission path of the current service request, and generating a topology reconfiguration strategy; generating cross-connection instructions based on the topology reconfiguration strategy and issuing them to the reconfigurable optical add-drop multiplexers for execution to complete the topology reconfiguration of the all-optical network.

[0006] In conjunction with the first aspect above, in one possible implementation, the aforementioned network operation status information includes node operation parameters of the reconfigurable optical add-drop multiplexer, transmission parameters of the optical fiber link, service request parameters, and overall network performance parameters. The method for obtaining the network operation status information of the all-optical network and constructing a multi-dimensional state vector specifically includes: obtaining the port occupancy rate, number of wavelengths added, number of wavelengths dropped, optical power loss, and electrical layer processing margin of the reconfigurable optical add-drop multiplexer to obtain node operation parameters; and obtaining the traffic load, number of wavelengths occupied, transmission loss, and crosstalk noise of the optical fiber link to obtain transmission parameters. Obtain the traffic demand, source node and destination node pair distribution, and number of incomplete connection requests for the current business request to obtain the business request parameters; obtain the overall network wavelength utilization, overall network blocking rate, and average transmission loss to obtain the overall network performance parameters; determine the local concentration of traffic, header traffic concentration, and traffic dispersion based on the source node and destination node pair distribution in the business request parameters; and fuse the node operating parameters and transmission parameters, the traffic demand and number of incomplete connection requests in the business request parameters, the local concentration of traffic, the header traffic concentration, the traffic dispersion, and the overall network performance parameters to construct a multi-dimensional state vector.

[0007] In conjunction with the first aspect mentioned above, in one possible implementation, the method for determining the optimal transmission path for the current service request based on the all-optical network physical topology, network operating status information, and multi-dimensional state vectors, through layer-by-layer decision-making in a hierarchical action space, includes: generating multiple candidate paths between source and destination nodes based on the all-optical network physical topology; determining the comprehensive path cost of each candidate path based on the number of intermediate nodes, port occupancy, optical power loss, and electrical layer processing margin in each candidate path, and determining the candidate path with the minimum comprehensive path cost as the optimal transmission path; determining the path optimization action of the optimal transmission path based on the all-optical network physical topology and node operating parameters; inputting the multi-dimensional state vector into a preset deep reinforcement learning network to determine the wavelength scheduling action of the intermediate nodes of the optimal transmission path; and determining the wavelength channel allocation result through a link-by-link verification and rollback mechanism based on the wavelength channel occupancy status and wavelength scheduling action of the fiber optic links in the optimal transmission path, thereby generating the link switching action.

[0008] In conjunction with the first aspect above, in one possible implementation, if there is a preceding service request before the current service request, the method for determining the comprehensive path cost of each candidate path further includes: determining a first correction coefficient based on the resource occupancy status of intermediate nodes in each candidate path according to the preceding service request; correcting the comprehensive path cost based on the first correction coefficient, so as to determine the optimal transmission path based on the corrected comprehensive path cost.

[0009] In conjunction with the first aspect above, in one possible implementation, if the current service request includes multiple service requests with the same start time, the method for determining the comprehensive path cost of each candidate path further includes: determining a second correction coefficient based on the number of common intermediate nodes among the candidate paths of the multiple current service requests and the traffic requirements of each service request; correcting the comprehensive path cost based on the second correction coefficient, so as to determine the optimal transmission path based on the corrected comprehensive path cost.

[0010] In conjunction with the first aspect above, in one possible implementation, the method for inputting a multidimensional state vector into a preset deep reinforcement learning network to determine the wavelength scheduling action of an intermediate node in the optimal transmission path specifically includes: generating a set of optional actions based on the wavelength occupancy status, port availability status, and link failure status of each node in the optimal transmission path; inputting the multidimensional state vector into the preset deep reinforcement learning network to calculate the action value function value of each candidate action in the set of optional actions; and determining the wavelength scheduling action of the intermediate node from the set of optional actions based on the action value function value.

[0011] In conjunction with the first aspect above, in one possible implementation, the method for determining the wavelength channel allocation result and generating link switching actions based on the wavelength channel occupancy status and wavelength scheduling actions of the fiber optic links in the optimal transmission path through a link-by-link verification and rollback mechanism specifically includes: marking the occupancy status of the wavelength channels of the fiber optic links in the optimal transmission path, and determining the set of common wavelength channels that are idle on the fiber optic links; if the common wavelength channel set is not empty, selecting a wavelength channel from the common wavelength channel set as the allocated wavelength channel, and verifying the port availability corresponding to the wavelength scheduling action node by node on the allocated wavelength channel, and allocating the verified allocated wavelength channel to the fiber optic links in the optimal transmission path; if the common wavelength channel set is empty, or the port availability verification fails, reselecting the optimal candidate path.

[0012] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: determining a global network performance evaluation value based on the overall network wavelength utilization, overall network blocking rate, and average transmission loss; determining a local action cost evaluation value based on the action type of wavelength scheduling actions, combined with the link lengths, wavelength occupancy, and cumulative transmission loss of the intermediate nodes in the optimal transmission path; constructing a multi-dimensional reward function that includes the global network performance evaluation value and the local action cost evaluation value, and updating the preset deep reinforcement learning network based on the multi-dimensional reward function.

[0013] In conjunction with the first aspect above, in one possible implementation, the method for determining the local concentration of traffic, the header traffic concentration, and the traffic dispersion based on the distribution of source and destination node pairs in the business request parameters specifically includes: determining the local concentration of traffic based on the ratio of the maximum number of requests for source and destination node pairs to the total number of requests; determining the header traffic concentration based on the ratio of the sum of the number of requests for source and destination node pairs that are ranked from largest to smallest within the first preset number to the total number of requests; and determining the traffic dispersion based on the probability distribution of source and destination node pairs.

[0014] Secondly, a reinforcement learning-based all-optical network topology reconfiguration system is provided, comprising: a topology construction module for acquiring the connection relationships between reconfigurable optical add-drop multiplexers and the link lengths of optical fiber links in the all-optical network, and constructing the physical topology of the all-optical network; a state awareness module for acquiring network operation state information of the all-optical network and constructing a multi-dimensional state vector; a hierarchical decision-making module for making decisions layer by layer in the hierarchical action space based on the physical topology of the all-optical network, the network operation state information, and the multi-dimensional state vector, determining the path optimization action, wavelength scheduling action, and link switching action corresponding to the optimal transmission path of the current service request, and generating a topology reconfiguration strategy; and a topology reconfiguration module for generating cross-connection instructions based on the topology reconfiguration strategy and issuing them to the reconfigurable optical add-drop multiplexers for execution, thereby completing the topology reconfiguration of the all-optical network.

[0015] The present invention has the following beneficial effects: By constructing a multi-dimensional state vector that integrates node operating parameters, transmission parameters, service request parameters, and overall network performance parameters, and designing a hierarchical action space that includes path optimization, wavelength scheduling, and link switching, and utilizing a deep reinforcement learning network for layer-by-layer decision-making, a comprehensive perception of the all-optical network's operating status and precise adaptation of multi-dimensional control capabilities are achieved. This effectively improves the matching degree between topology reconstruction decisions and dynamic service scenarios, reduces network congestion rate, improves wavelength resource utilization efficiency, and optimizes transmission loss, thus realizing efficient topology reconstruction of the all-optical network under dynamic traffic scenarios. Attached Figure Description

[0016] Figure 1 A system architecture diagram of an all-optical network topology reconstruction system based on reinforcement learning is provided in one embodiment of the present invention; Figure 2 A flowchart illustrating a reinforcement learning-based all-optical network topology reconstruction method according to an embodiment of the present invention; Figure 3 This is a network architecture diagram of an all-optical network provided in one embodiment of the present invention. Detailed Implementation

[0017] The following description, in conjunction with the accompanying drawings, details a specific scheme for a reinforcement learning-based all-optical network topology reconstruction method and system provided by this invention.

[0018] Please see Figure 1 The diagram illustrates a system architecture of an all-optical network topology reconstruction system based on reinforcement learning, according to an embodiment of the present invention. The all-optical network topology reconstruction system based on reinforcement learning includes: a topology construction module 1, a state awareness module 2, a hierarchical decision module 3, and a topology reconstruction module 4.

[0019] Among them, the topology construction module 1 is used to obtain the connection relationship between reconfigurable optical add-drop multiplexers in the all-optical network and the link length of the optical fiber link, and construct the physical topology of the all-optical network. State awareness module 2 is used to acquire network operation state information of the all-optical network and construct a multi-dimensional state vector; The hierarchical decision module 3 is used to make decisions layer by layer in the hierarchical action space based on the physical topology of the all-optical network, network operation status information and multi-dimensional state vectors, to determine the path optimization action, wavelength scheduling action and link switching action corresponding to the optimal transmission path of the current service request, and to generate a topology reconstruction strategy. Topology reconfiguration module 4 is used to generate cross-connect instructions according to the topology reconfiguration strategy and send them to the reconfigurable optical add-drop multiplexer for execution, so as to complete the topology reconfiguration of the all-optical network.

[0020] Please see Figure 2 The diagram illustrates a flowchart of a reinforcement learning-based all-optical network topology reconstruction method according to an embodiment of the present invention. This reinforcement learning-based all-optical network topology reconstruction method includes: S1. Obtain the connection relationships between reconfigurable optical add-drop multiplexers in the all-optical network and the link length of the optical fiber links, and construct the physical topology of the all-optical network.

[0021] An all-optical network refers to an optical communication network architecture where the transmission, switching, and processing of service signals are all completed within the optical domain. Its core network element is the reconfigurable optical add-drop multiplexer (OCD). The OCD is a node in the all-optical network, employing a multi-directional structure, supporting add-drop and cross-scheduling of arbitrary wavelengths, and enabling switching of arbitrary wavelengths between different nodes through wavelength selection switches. Connection relationships refer to the physical layer connectivity established between reconfigurable OCDs via fiber optic links, characterizing which node pairs in the network have direct fiber optic link connections. Link length refers to the physical transmission distance of the fiber optic link connecting two reconfigurable OCDs; this parameter directly relates to the transmission loss and delay of optical signals on the link. Physical topology refers to an undirected graph that abstracts reconfigurable OCDs as graph nodes, fiber optic links as edges, and uses link length as the edge weight, used to formally describe the physical connection structure of the all-optical network. For example... Figure 3 As shown, the all-optical network adopts a mesh topology commonly used in backbone networks, including multiple reconfigurable optical add-drop multiplexer nodes 102 and multiple optical fiber links connecting each node. Each optical fiber link 103 supports bidirectional transmission and includes N wavelength channels 101. Figure 3 For example, wavelength division multiplexing (WDM) technology is used to multiplex 128 wavelength channels 101 within each optical fiber link 103, and each wavelength channel 101 can independently carry service traffic at a rate of 100Gbps.

[0022] In some implementations, the node identifiers and port configuration information of each reconfigurable optical add-drop multiplexer (OPD) are read from the network management system of the all-optical network or a pre-built network configuration database. The port records of each node are traversed to identify the optical fiber links connecting different nodes, extract the node numbers of the two OPDs associated with each optical fiber link, and record the attribute that the link supports bidirectional transmission.

[0023] For each identified fiber optic link, extract its physical length attribute from network planning data or physical measurement records. The link length can be the actual fiber optic laying distance or the equivalent transmission distance measured by an optical time domain reflectometer.

[0024] Each reconfigurable optical add-drop multiplexer (OAD) node is considered a node in the physical topology, and each fiber optic link is considered an edge connecting two nodes. Each edge records the identifiers of the two connected vertices and the length of the fiber optic link, forming an undirected graph as a mathematical model of the all-optical network's physical topology. The vertex set of this undirected graph records the location distribution of all reconfigurable OADs in the all-optical network, and the edge set records the fiber optic connections between nodes and the distance information of each link.

[0025] S2. Obtain the network operation status information of the all-optical network and construct a multi-dimensional state vector.

[0026] Network operational status information refers to a multi-dimensional data set collected in real-time from an all-optical network environment by a pre-defined deep reinforcement learning network. This data characterizes the current operational status of the network, including node operating parameters of reconfigurable optical add-drop multiplexers (ROD+DDLs), transmission parameters of fiber optic links, service request parameters, and overall network performance parameters. Node operating parameters characterize the internal resource usage and physical state of each ROD+DDL node; transmission parameters characterize the resource load and transmission quality of each fiber optic link; service request parameters characterize the demand characteristics and spatiotemporal distribution of currently arriving services; and overall network performance parameters characterize the overall operational health of the all-optical network. The multi-dimensional state vector is a high-dimensional continuous vector formed by fusing the parameters from the above four levels after preprocessing. This vector transforms heterogeneous network data into a numerical representation with a unified scale, which is then input into the pre-defined deep reinforcement learning network to support subsequent topology reconfiguration decisions.

[0027] In some implementations, the port occupancy rate, number of wavelengths added, number of wavelengths dropped, optical power loss, and electrical layer processing margin of the reconfigurable optical add-drop multiplexer (ODD) are obtained to obtain the node's operating parameters. Port occupancy rate represents the proportion of physical ports of the current node that are occupied by services; the number of wavelengths added represents the number of wavelength services injected into the optical network by the current node through added ports; the number of wavelengths dropped represents the number of wavelength services received from the optical network by the current node through dropped ports; optical power loss represents the degree of power attenuation of the optical signal as it passes through the node; and electrical layer processing margin represents the remaining usable proportion of the node's electrical layer conversion and regeneration capabilities. Specifically, by calling the data acquisition interface of the all-optical network model, the port occupancy rate, number of wavelengths added, number of wavelengths dropped, optical power loss, and electrical layer processing margin are read from the status record data structure of each ODD. Because the dimensions and numerical ranges of the above parameters differ, normalization is required to combine different types of data. For example, a maximum value normalization method can be used. For the same parameter type, the values ​​of the parameter are traversed across all reconfigurable optical add-drop multiplexer nodes in the current all-optical network, and the maximum value is taken as the normalization denominator. The original value of the parameter for each node is divided by this maximum value to obtain the normalized value of the corresponding node. Through this normalization process, the node-level normalized characteristics of each node are obtained, which are used to characterize the relative resource occupancy level of the node.

[0028] Transmission parameters are obtained by acquiring the traffic load, wavelength occupancy, transmission loss, and crosstalk noise of the fiber optic link. Traffic load represents the total traffic currently flowing through the fiber optic link; wavelength occupancy represents the number of wavelength channels in the fiber optic link that are currently occupied by traffic; transmission loss represents the power attenuation of the optical signal during transmission on the fiber optic link; and crosstalk noise represents the noise power level introduced by crosstalk between wavelength channels. Specifically, traffic load, wavelength occupancy, transmission loss, and crosstalk noise are read from the status record data structure of each fiber optic link. Since the dimensions and numerical ranges of these parameters differ, normalization processing is required to combine different types of data. For example, a maximum value normalization method is used. For the same parameter type, the value of this parameter is traversed across all fiber optic links in the current all-optical network, and the maximum value is taken as the normalization denominator. The original value of this parameter for each link is divided by this maximum value to obtain the normalized value of the corresponding link. Through this normalization processing, the link-level normalized characteristics of each link are obtained, which are used to characterize the relative load level of link resources.

[0029] The traffic demand, source-destination node pair distribution, and number of incomplete connection requests for the current service request are obtained to obtain the service request parameters. Traffic demand represents the transmission bandwidth required by the currently arriving service request; source-destination node pair distribution represents the spatial distribution of existing or arrived connection requests between each source node and each destination node in the current network; and the number of incomplete connection requests represents the total number of connection requests that have not yet been released and are still in an occupied state in the current network. Specifically, the traffic demand, source-destination node pair distribution, and number of incomplete connection requests for the current service request are read from the service request queue. Then, the traffic demand and the number of incomplete connection requests are normalized respectively. The traffic demand of the current service request is divided by the theoretical maximum traffic demand to obtain the normalized value of the traffic demand, and the number of incomplete connection requests is divided by the total number of source-destination node pairs with all recorded connection requests in the current network to obtain the normalized value of the number of incomplete connection requests. The theoretical maximum traffic demand is preset based on the maximum service rate that a single wavelength channel can carry in the all-optical network; in this embodiment, it can be set to 50Gbps.

[0030] The distribution of source and destination nodes can be represented by a two-dimensional matrix. The number of rows and columns of the matrix are equal to the total number of reconfigurable optical add-drop multiplexer nodes in the all-optical network. For example, the rows of the matrix are source nodes and the columns are destination nodes. The diagonal elements are all 0, and the off-diagonal elements represent the number of connection requests that are currently being transmitted or have arrived for each source and destination node pair.

[0031] Since the matrix has a high dimension and directly inputting it into a preset deep reinforcement learning network would result in an excessively large dimension, it is necessary to determine the local concentration of traffic, the concentration of head traffic, and the dispersion of traffic based on the distribution of source and destination nodes in the business request parameters. That is, the above three feature values ​​are extracted from the matrix to characterize the distribution characteristics of the matrix.

[0032] Local concentration of traffic is used to characterize the degree of peak concentration of traffic in the spatial distribution of a network. Specifically, it is determined by the ratio of the maximum number of requests between the source and destination nodes in the matrix to the total number of requests.

[0033] Header traffic concentration is used to characterize the degree of concentration of the main traffic flow in the network. Specifically, it is determined by the ratio of the sum of the number of requests of the source node and destination node pairs in the matrix from largest to smallest (the first 3 in this embodiment) to the total number of requests.

[0034] Traffic dispersion is used to characterize the degree of spatial uniformity of traffic distribution in a network. Specifically, it is determined based on the probability distribution of source and destination node pairs. The process involves iterating through all off-diagonal elements of the matrix, dividing the number of requests for each element by the total number of requests to obtain the probability value for each off-diagonal element. The sum of the probability values ​​for all off-diagonal elements is 1. According to the Shannon entropy principle, the distribution entropy is calculated based on the probability values ​​of each off-diagonal element. This is achieved by calculating the base-2 logarithm of each probability value, multiplying it by the probability value, taking the negative sign, and summing the results for all probability values. The calculated distribution entropy is then divided by the theoretical maximum entropy value to obtain the traffic dispersion. The theoretical maximum entropy value is determined by the total number of off-diagonal elements in the matrix, which is the square of the total number of nodes minus the total number of nodes. For example, in a 10×10 matrix, the total number of off-diagonal elements is 10×10-10=90, and the theoretical maximum entropy is the base-2 logarithm of 90. Specifically, when the probability of a certain off-diagonal element is 0, no logarithmic operation is performed; its contribution to the distribution entropy is directly set to 0, and the distribution entropy is obtained by summing the results of calculations of other probability values.

[0035] The overall network performance parameters are obtained by acquiring the overall wavelength utilization rate, overall network congestion rate, and average transmission loss. The overall wavelength utilization rate represents the proportion of all wavelength channels in the current all-optical network that are occupied by services. The overall network congestion rate represents the proportion of service requests rejected due to insufficient resources in the current all-optical network. The average transmission loss represents the average transmission loss of all established optical paths in the current all-optical network. Specifically, the overall wavelength utilization rate, overall network congestion rate, and average transmission loss are read from the overall network statistical data structure. The wavelength utilization rate and congestion rate are normalized values; normalization is achieved by dividing the average transmission loss by the maximum tolerable loss of the system. The maximum tolerable loss of the system is determined by the link budget during the all-optical network system design. This normalization method ensures that the average transmission loss value accurately reflects how close the current overall network transmission quality is to the system's tolerance limit. Specifically, if the average transmission loss exceeds the maximum tolerable loss of the system, the normalized result is truncated to 1.

[0036] A multi-dimensional state vector is constructed by fusing node operating parameters and transmission parameters, traffic demand and number of incomplete connection requests in service request parameters, traffic local concentration, header traffic concentration, traffic dispersion, and overall network performance parameters. Specifically, the normalized values ​​of port occupancy, wavelength addition, wavelength drop, optical power loss, and electrical layer processing margin of all reconfigurable optical add-drop multiplexers are arranged sequentially to obtain a node-level feature sequence; the normalized values ​​of traffic load, wavelength occupancy, transmission loss, and crosstalk noise of all fiber links are arranged sequentially to obtain a link-level feature sequence; the normalized values ​​of traffic demand, number of incomplete connection requests, traffic local concentration, header traffic concentration, and traffic dispersion of the current service request are arranged sequentially to obtain a service-level feature sequence; the normalized value of average transmission loss, overall network wavelength utilization, and overall network blocking rate are arranged sequentially to obtain a network-wide feature sequence; and the above four feature sequences are concatenated sequentially to form a one-dimensional numerical sequence as a multi-dimensional state vector. The dimensions of the multidimensional state vector are dynamically adjusted according to the scale of the all-optical network. The calculation rules are as follows: the total number of nodes multiplied by the number of node-level feature dimensions, the total number of links multiplied by the number of link-level feature dimensions, and the sum of the number of service request feature dimensions and the number of network-wide feature dimensions.

[0037] Specifically, when normalizing each parameter using the maximum value normalization method, if the global maximum value of a certain parameter in the all-optical network is 0, indicating that the network is currently in a completely unloaded or initial operating state, then the normalization ratio calculation for that parameter type is skipped, and the normalized value is directly assigned to 0.

[0038] S3. Based on the physical topology of the all-optical network, network operation status information, and multi-dimensional state vectors, make decisions layer by layer in the hierarchical action space to determine the path optimization action, wavelength scheduling action, and link switching action corresponding to the optimal transmission path of the current service request, and generate a topology reconstruction strategy.

[0039] A layered action space refers to a three-layered progressive action set formed by decoupling the multi-dimensional control capabilities of reconfigurable optical add-drop multiplexers at the path, wavelength, and port granularities. Its purpose is to reduce the decision complexity of traditional flat action spaces and improve the flexibility of topology reconfiguration. Path optimization action refers to the decision to select the optimal transmission path for the current service request based on the physical topology of the all-optical network. This action determines the end-to-end route of the service flow at the path granularity. Wavelength scheduling action refers to the decision to determine the processing method for a specific wavelength at each node in the optimal transmission path: cut-through, add, or discard. The source node performs wavelength addition, the destination node performs wavelength discard, and intermediate nodes perform wavelength cut-through. This action achieves precise scheduling of wavelength resources within a node at both the wavelength and port granularities. Link switching action refers to the decision to complete the conflict-free allocation of wavelength channels along the entire path and control link connectivity based on the wavelength scheduling action and the wavelength channel occupancy status of the fiber optic link. This action achieves final confirmation and resource occupancy of the wavelength channel at the link granularity. Topology reconfiguration strategy refers to a complete decision-making scheme formed by the combination of the above three layers of actions, used to guide reconfigurable optical add-drop multiplexers to perform cross-connection configuration to complete the all-optical network topology reconfiguration. Layer-by-layer decision-making in the hierarchical action space involves narrowing the candidate solution space through the path optimization layer, filtering physically feasible actions through the wavelength scheduling layer, and verifying full-path consistency through the link switching layer. This reduces the action complexity from a product form of direct, flattened control to a summation form of hierarchical control, improving decision-making efficiency and network performance.

[0040] In the hierarchical decision-making process within the layered action space, the path optimization action layer is executed first. Based on the physical topology of the all-optical network, multiple candidate paths between the source and destination nodes are generated. The comprehensive path cost of each candidate path is determined based on the number of intermediate nodes, port occupancy, optical power loss, and electrical layer processing margin. The candidate path with the lowest comprehensive path cost is identified as the optimal transmission path, and the sequence of all nodes traversed by this path is output as the path optimization action. Next, the wavelength scheduling action layer is executed. For each intermediate node in the optimal transmission path output by the path optimization action layer, a multi-dimensional state vector is input into a pre-defined deep reinforcement learning network. The pre-defined deep reinforcement learning network calculates the action value function value for each candidate action. Based on the action value function value, the wavelength scheduling action for that intermediate node is determined, and the wavelength scheduling action for each intermediate node, along with its corresponding wavelength number and port number, is output as the wavelength scheduling action. Finally, the link switching action layer is executed. For the wavelength numbers involved in the wavelength scheduling actions of each intermediate node output by the wavelength scheduling action layer, the idle status of the corresponding wavelength channels on each fiber optic link in the optimal transmission path is detected. A common set of wavelength channels that are idle on all fiber optic links is determined, and a wavelength channel is selected from this set as the allocated wavelength channel. The allocation result is then output as the link switching action. The decision results from these three layers are combined to generate the topology reconstruction strategy.

[0041] In some implementations, multiple candidate paths between source and destination nodes are generated based on the physical topology of the all-optical network. Specifically, in the path optimization action layer, an undirected graph corresponding to the physical topology of the all-optical network is obtained. Using the vertex and edge sets of this undirected graph as input, and the source and destination node identifiers of the current service request as query conditions, the K shortest paths (K-shortest paths, KSP) algorithm is used to calculate the top K shortest paths from the source node to the destination node as a set of candidate paths. The value of K is dynamically set according to the scale of the all-optical network. For example, when the network scale is large (more than 50 nodes and more than 100 links), and more candidate paths are needed to cover diverse path selections, the value of K is taken as the rounded-up value of one-fifth of the total number of fiber links. When the network scale is limited, such as in this embodiment where the all-optical network includes 10 reconfigurable optical add-drop multiplexers and 15 fiber links, the value of K is taken as the rounded-up value of one-tenth of the total number of fiber links.

[0042] In addition to being uniformly set according to the scale of the all-optical network, the K value can also be dynamically adjusted based on the local topology density of the source node. If the source node and the destination node are adjacent nodes, and the source node has only one adjacent node, the physical layer has severely constrained path diversity due to the very limited number of selectable first-hop links. In this case, it is not necessary to generate multiple candidate paths through KSP, and the single path can be directly used as the optimal path. If the source node and the destination node are adjacent nodes, but the source node has other adjacent nodes besides the destination node, and there are other detour routes between the source node and the destination node, the path selection freedom is relatively high. In this case, the K value can be determined proportionally according to the number of adjacent nodes of the source node. For example, in the case of medium topology density (the number of adjacent nodes of the source node is less than 50% of the total number of nodes in the all-optical network), the K value is taken as one-third of the number of adjacent nodes, rounded up. In this case, a smaller proportion is used to avoid generating redundant candidate paths. In the case of dense topology region (the number of adjacent nodes of the source node is greater than or equal to 50% of the total number of nodes in the all-optical network), the K value is taken as half of the number of adjacent nodes, rounded up. In this case, a larger proportion is used to make full use of the highly interconnected path selection space.

[0043] Based on the number of intermediate nodes, port occupancy, optical power loss, and electrical layer processing margin in each candidate path, the overall path cost of each candidate path is determined, and the candidate path with the minimum overall path cost is identified as the optimal transmission path. The overall path cost is a numerical indicator used to quantitatively evaluate the transmission quality of a candidate path; a smaller value indicates better transmission quality. The number of intermediate nodes refers to the number of forwarding nodes in the candidate path excluding the source and destination nodes. This number directly affects the cumulative transmission loss of the optical signal and the number of wavelength additions and drops. Specifically, the number of intermediate nodes in each candidate path and the total number of intermediate nodes in all candidate paths are counted, and the ratio of the number of intermediate nodes in each candidate path to the total number of intermediate nodes in all candidate paths is calculated. Simultaneously, since the electrical layer processing margin characterizes the sufficiency of node processing capacity, a larger normalized value indicates more available resources. Subtracting the normalized value of the electrical layer processing margin from 1 yields the electrical layer processing margin cost mapping value. The mean of the normalized values ​​of port occupancy, optical power loss, and the electrical layer processing margin cost mapping value is then calculated. Then, the ratios and means mentioned above are weighted and summed. Since the mean directly reflects the current resource occupancy status and signal processing capability of each intermediate node, its impact on transmission quality is more direct, and its weight is greater than that of the ratio. For example, the weight of the ratio is set to 0.4, and the weight of the mean is set to 0.6. Specifically, if there is only one direct path between the source node and the destination node, and no intermediate nodes exist, the comprehensive path cost corresponding to the direct path is set to 0. Through the above calculations, the comprehensive path cost of each candidate path is obtained, and the candidate path with the minimum value is selected as the optimal transmission path, which achieves a balance between node resource status and path hop count.

[0044] Furthermore, if a preceding service request exists before the current service request—that is, if an existing service request is currently being processed in the all-optical network—and this preceding service request has already occupied some intermediate node resources, resource contention and congestion risks may occur if these occupied intermediate nodes are traversed again by the candidate path of the current service request. In this case, a first correction coefficient is determined based on the resource occupancy status of intermediate nodes in each candidate path according to the preceding service request. Then, the overall path cost is corrected based on the first correction coefficient to determine the optimal transmission path. Specifically, for each candidate path, the number of first nodes occupied by preceding service requests and the number of second nodes not occupied in the intermediate nodes of that candidate path are counted. The first correction coefficient is calculated by weighted summation of the ratio of the number of first nodes to the total number of intermediate nodes and the ratio of the number of second nodes to the total number of intermediate nodes. The weighting of the first node is higher than that of the second node because when an intermediate node in the candidate path is occupied by a preceding service request, the probability of resource contention increases, requiring a higher penalty coefficient. For example, the weighting of the first node is set to 0.6, and the weighting of the second node is set to 0.4. The overall path cost is corrected based on a first correction coefficient. The sum of 1 and the first correction coefficient is used as a first amplification coefficient and multiplied by the overall path cost to obtain the corrected overall path cost. The candidate path with the lowest corrected overall path cost is determined as the optimal transmission path.

[0045] If the current service request includes multiple service requests with the same start time (i.e., multiple service requests arrive simultaneously and require parallel processing), there may be common intermediate nodes between the candidate paths of these service requests. When multiple service requests simultaneously choose different candidate paths passing through the same intermediate node, that intermediate node needs to process multiple services concurrently, significantly increasing the risk of congestion. In this case, a second correction coefficient is determined based on the number of common intermediate nodes between the candidate paths of the multiple service requests and the traffic demand of each service request. The comprehensive path cost is then adjusted based on this second correction coefficient to determine the optimal transmission path. Specifically, for each candidate path of each service request with common intermediate nodes, the number of common intermediate nodes between this candidate path and the candidate paths of other service requests is counted. The number of common intermediate nodes is divided by the total number of intermediate nodes in this candidate path to obtain the proportion of common intermediate nodes. The proportion of common intermediate nodes is then weighted and summed with the normalized value of the traffic demand of this service request to calculate the second correction coefficient. An increase in the number of common intermediate nodes and an increase in traffic demand both non-linearly increase the risk of congestion; therefore, they can be set to the same weight of 0.5. The path cost is adjusted based on the second correction factor. The sum of the value 1 and the second correction factor is used as a second amplification factor and multiplied by the path cost to obtain the adjusted path cost. The candidate path with the minimum adjusted path cost is determined as the optimal transmission path.

[0046] Optionally, the first correction coefficient and the second correction coefficient can be used together. If there is a preceding business request before the current business request, and the current business request includes multiple business requests with the same start time, then the second amplification coefficient is multiplied by the path comprehensive cost after correction by the first correction coefficient to obtain the path comprehensive cost after secondary correction. At this time, the candidate path with the smallest path comprehensive cost has the highest priority.

[0047] Specifically, if there is only one direct path between the source node and the destination node, with no intermediate nodes and no source of resource contention risk, the overall path cost will not be included in the correction factor.

[0048] The path optimization action determines the optimal transmission path based on the physical topology of the all-optical network and the node operating parameters. The path optimization action refers to using the optimal transmission path determined in the aforementioned hierarchical decision-making process as the end-to-end routing scheme for the current service request. Specifically, the node sequence and link sequence of the optimal transmission path are encapsulated into a path optimization action data structure. This data structure includes source node identifiers, destination node identifiers, intermediate node sequences, and the sequence of traversed fiber links, used to identify the complete transmission path of the service flow in the physical topology.

[0049] The multidimensional state vector is input into a pre-defined deep reinforcement learning network to determine the wavelength scheduling actions of intermediate nodes in the optimal transmission path. The wavelength scheduling action refers to determining the processing method for a specific wavelength for each node in the optimal transmission path, namely, pass-through, add, or discard, and initially filtering the decision results based on the corresponding inlet port number and outlet port number.

[0050] Specifically, a set of optional actions is generated based on the wavelength occupancy status, port availability status, and link failure status of each node in the optimal transmission path. Wavelength cut-through, wavelength addition, and wavelength discarding are three basic operation types of reconfigurable optical add-drop multiplexers in all-optical networks at the wavelength granularity. Wavelength cut-through refers to a node directly cross-connecting the input signal from the input port to the output port without performing add-on or add-off operations, allowing the signal to continue transmitting along the fiber link to the next node. This action only involves optical layer configuration, minimizes signal delay, and does not perform electrical layer regeneration. Wavelength addition refers to a node inputting local service traffic into the all-optical network through an add port, enabling that wavelength channel to carry locally originating services and continue transmission along the output port direction. Wavelength discarding refers to a node extracting a specific wavelength signal from the input port direction from the all-optical network through a discard port and outputting it locally, allowing the service traffic to reach the destination node. For the current end-to-end service request, the source node only performs the wavelength addition action, the destination node only performs the wavelength discard action, and intermediate nodes only perform the wavelength cut-through action. Determining the feasibility of the three wavelength scheduling actions requires a comprehensive assessment of each node's wavelength occupancy status, port availability status, and link fault status. The wavelength pass-through action requires both the ingress and egress ports to be in normal working order and not fully occupied; the wavelength frequency positions in the ingress and egress directions to be consistent; the wavelength not being occupied by other services in either the ingress or egress direction; and the fiber optic link connected to the egress port to be fault-free. The wavelength addition action requires the adding port to be in normal working order and not fully occupied; the wavelength not being occupied by other services in the egress direction; and the fiber optic link connected to the egress port to be fault-free. The wavelength discard action requires the discarding port to be in normal working order and not fully occupied; the wavelength not being occupied by other services in the ingress direction; and the fiber optic link connected to the ingress port to be fault-free. The candidate actions retained after the above determination constitute the optional action set. If the optional action set is empty, a current path unavailable signal is returned to the path optimization action layer, and a suboptimal path is reselected as the optimal transmission path before re-entering the wavelength scheduling action layer for processing.

[0051] The multidimensional state vector is input into a pre-defined deep reinforcement learning network to calculate the action value function value of each candidate action in the available action set. The pre-defined deep reinforcement learning network is a deep Q-network, which includes an input layer, three fully connected hidden layers, and an output layer. The dimension of the input layer is the same as the dimension of the multidimensional state vector, defining all possible action combinations for wavelength scheduling that the reconfigurable optical add-drop multiplexer can perform. The dimension of the output layer is fixed to the maximum number of candidate actions supported by the system. The three fully connected hidden layers are used to perform nonlinear transformations on the input multidimensional state vector to extract high-dimensional features. The output layer outputs the action value function value corresponding to each candidate action. Since the number of available actions is usually less than the maximum number of candidate actions, a masking mechanism is introduced: unavailable actions not in the available action set are identified, and their action value function values ​​are overwritten with a very small negative penalty value (e.g., -10). 9 Therefore, when selecting wavelength scheduling actions based on action value function values, only actions that are objectively feasible at the current intermediate node are selected, resulting in action value function values ​​for each candidate action. The action value function value is used to evaluate the cumulative expected reward that can be obtained from executing the candidate action in the current network state in the long run; the larger the value, the higher the long-term benefit of executing the action.

[0052] Since wavelength pass-through refers to intermediate nodes directly forwarding signals from the input port to the output port without performing electrical layer regeneration, the cost of this action increases with the transmission distance. It also consumes the available resources of one wavelength channel in each of the preceding and following fiber optic links, increasing the risk of congestion, and signal quality decreases with the cumulative transmission distance. Therefore, the cost function value for intermediate nodes performing wavelength pass-through needs to be adjusted. Specifically, the first cost parameter is obtained by dividing the sum of the forward and backward link lengths of the intermediate node by the total length of the optimal transmission path. The ratio of the number of occupied wavelengths to the total number of wavelength channels for the forward and backward links is calculated, and the average of these two ratios is used as the second cost parameter. The ratio of the cumulative transmission loss from the source node to the next intermediate node to the maximum tolerable loss of the system is used as the third cost parameter. The maximum tolerable loss is determined by the link budget in the all-optical network system design and is a preset fixed value. Specifically, it equals the difference between the output optical power of the entire network's optical transmitter and the minimum sensitivity of the optical receiver, minus a preset system margin. In this embodiment, based on system design experience at a transmission rate of 100Gbps, this value is preset to 25dB. Then, the average of the first, second, and third cost parameters is used as the limit value for the pass-through action at that wavelength. The action value function output by the preset deep reinforcement learning network is corrected based on this limit value. The limit value is multiplied by a limit coefficient (ranging from 0.3 to 0.5; in this embodiment, 0.4 can be used) to prevent over-adjustment of the action value function value by the limit value. Finally, the product of the limit value and the limit coefficient is subtracted from the value 1 to obtain the correction coefficient. When the action value function output by the preset deep reinforcement learning network is positive, the correction coefficient is multiplied by the action value function output by the preset deep reinforcement learning network to obtain the corrected action value function value; when the output action value function value is negative or zero, no correction is made, and the action value function value is directly used as the corrected action value function value.

[0053] For wavelength scheduling actions of wavelength addition and wavelength discard, no modification is made, and the original action value function value output by the preset deep reinforcement learning network is used directly.

[0054] The wavelength scheduling action for intermediate nodes is determined from the set of available actions based on the action value function value. Specifically, a pre-defined greedy strategy can be used to select the wavelength scheduling action for the current intermediate node from the set of available actions: the exploration rate ε gradually decreases from 1.0 to 0.01 with each training round, with a decay step count of 5000 steps. That is, after each decision step, ε decreases by (1.0-0.01) / 5000 until it reaches 0.01 and then remains constant. The candidate action with the largest action value function value is selected as the wavelength scheduling action with a probability of 1-ε. At the same time, a candidate action is randomly selected from the set of available actions with a probability of ε. This candidate action is used to train the deep reinforcement learning network to learn the differentiated reward feedback brought by different actions in the same state, thereby gradually correcting the estimation bias of the action value function value, avoiding premature convergence of the network to a local optimum, and enhancing the ability to explore unknown actions.

[0055] Based on the wavelength channel occupancy status and wavelength scheduling actions of the fiber optic links in the optimal transmission path, the wavelength channel allocation result is determined through a link-by-link verification and rollback mechanism, generating a link switching action. The link switching action refers to the conflict-free allocation of wavelength channels along the entire path through link-by-link verification based on wavelength scheduling actions and the wavelength channel occupancy status of the fiber optic links, and the execution of rollback and reselection decisions when verification fails.

[0056] Specifically, the occupancy status of wavelength channels in the fiber optic links of the optimal transmission path is marked to determine the set of common wavelength channels that are idle on the fiber optic links. For each fiber optic link in the optimal transmission path, the occupancy status of all 128 wavelength channels in each fiber optic link is obtained. Wavelength channels that are occupied are marked as 1, and wavelength channels that are idle are marked as 0. These are arranged in order from 1 to 128, resulting in a 128-length marker vector for each fiber optic link. A logical OR operation is performed on the elements at the same position in the marker vectors of all fiber optic links to obtain a new 128-length vector. In this vector, a value of 1 indicates that at least one fiber optic link is occupied on that wavelength channel, and a value of 0 indicates that all fiber optic links are idle on that wavelength channel. Thus, the wavelength channels corresponding to the positions with a value of 0 constitute the common wavelength channel set.

[0057] If the common wavelength channel set is not empty, a wavelength channel is selected from the common wavelength channel set as the allocated wavelength channel. The selection method is to calculate the occupancy rate of each wavelength channel across the entire network (i.e., global occupancy rate), and prioritize the wavelength channel with the lowest global occupancy rate to achieve load balancing. The port availability corresponding to the wavelength scheduling action is verified node by node on the allocated wavelength channel: for the source node, it is verified whether its added port is idle and available; for the destination node, it is verified whether its discarded port is idle and available; for intermediate nodes, it is verified whether both its input and output ports are in normal working condition and not fully occupied, and it is confirmed that the wavelength frequency positions in the input and output port directions are consistent. If the port verification of all nodes passes, a wavelength-level configuration command is issued to the wavelength selection switch of each reconfigurable optical add-drop multiplexer, allocating the verified allocated wavelength channel to the fiber optic link in the optimal transmission path.

[0058] If the common wavelength channel set is empty, or port availability verification fails, a new optimal candidate path is selected. Specifically, if the common wavelength channel set is empty, it means there are currently no wavelength channels that can be allocated without conflict along the entire optimal transmission path. In this case, a "current path unavailable" signal is returned to the path optimization action layer. The path optimization action layer then selects the candidate path with the second-lowest overall path cost from the candidate path set as the optimal transmission path, and the process proceeds again to the wavelength scheduling action layer and the link switching action layer. If the common wavelength channel set is not empty, but port verification fails or wavelength selection switch configuration fails at any node, only the successfully configured node segment is rolled back, the occupied wavelength resources are released, and a "current path unavailable" signal is returned to the path optimization action layer. This triggers the selection of the candidate path with the second-lowest overall path cost as the optimal transmission path, and the process proceeds again to the wavelength scheduling action layer and the link switching action layer.

[0059] Furthermore, if the number of reselected paths reaches the total number of paths in the candidate path set, and all candidate paths cannot be allocated due to the public wavelength channel set being empty or port availability verification failing, then the reselection of the current service request is terminated, the current service request blocking event is reported to the network management system, the port and wavelength resource records temporarily occupied by this service request during the wavelength scheduling process are released, and the current topology reconstruction process ends.

[0060] If all configurations are successful, the entire network resource status table will be updated synchronously, and the allocated wavelength channel on all fiber optic links in the optimal transmission path will be marked as occupied.

[0061] The allocated wavelength channel number of each fiber link is output by the link switching action layer, and combined with the optimal transmission path node sequence output by the path optimization action layer and the wavelength scheduling action and port number of each intermediate node output by the wavelength scheduling action layer to form the topology reconstruction strategy.

[0062] Furthermore, during the layer-by-layer decision-making process, the pre-defined deep reinforcement learning network needs to continuously optimize its network parameters through training. After a complete reconstruction decision of the all-optical network is completed, the global network performance evaluation value is determined based on the overall network wavelength utilization, overall network blocking rate, and average transmission loss. Specifically, for the multi-objective optimization requirements of all-optical network topology reconstruction, a weighted reward function is constructed, which needs to comprehensively consider three core indicators: resource utilization, blocking rate, and transmission loss. Among them, resource utilization is positively correlated with overall network wavelength utilization; the higher the overall network wavelength utilization, the larger the reward value, which is used to guide the pre-defined deep reinforcement learning network to improve resource utilization efficiency. The blocking rate reward is negatively correlated with overall network blocking rate; the lower the overall network blocking rate, the larger the reward value, which is used to guide the pre-defined deep reinforcement learning network to reduce the service request blocking rate. The transmission loss reward is negatively correlated with average transmission loss; the lower the average transmission loss, the larger the reward value, which is used to guide the pre-defined deep reinforcement learning network to optimize transmission paths and reduce optical layer transmission loss. Therefore, the average of overall network wavelength utilization, (1-overall network blocking rate), and (1-average transmission loss / maximum tolerable loss of the system) is used as the global network performance evaluation value.

[0063] Based on the action type of wavelength scheduling, and considering the link lengths, wavelength occupancy, and cumulative transmission loss of the intermediate nodes in the optimal transmission path, a local action cost evaluation value is determined. For intermediate nodes performing wavelength pass-through, the local action cost value is calculated as 1 minus the limit value of the pass-through action. The average of the local action cost values ​​of all intermediate nodes is calculated to obtain the overall local action cost evaluation value. If the optimal transmission path is a direct path with no intermediate nodes, the local action cost evaluation value is directly assigned to 1, without additional penalty. Through this local evaluation, pass-through actions that lead to continuous wavelength occupancy and signal quality degradation are dynamically penalized.

[0064] A multi-dimensional reward function is constructed, comprising a global network performance evaluation value and a local action cost evaluation value. The preset deep reinforcement learning network is then updated based on this multi-dimensional reward function. Specifically, the weighted sum of the global network performance evaluation value and the local action cost evaluation value can be used as the total reward function. The global network performance evaluation value is the final evaluation metric for the entire network, and its weight should be greater than that of the local action cost evaluation value; for example, the weight of the global network performance evaluation value is 0.7, and the weight of the local action cost evaluation value is 0.3. The reward value calculated by the multi-dimensional reward function is used as the training target value of the preset deep reinforcement learning network. A preset mean squared error loss function is used to calculate the error between the target value and the action value function output by the preset deep reinforcement learning network. The network parameters of the preset deep reinforcement learning network are updated using the backpropagation algorithm. The experience pool capacity of the preset deep reinforcement learning network is 10000. During each training iteration, 64 experience data points are randomly sampled from the experience pool as a batch for training. The target network updates its parameters every 100 steps, with a discount factor of 0.95 and a learning rate of 0.001. Through the above training process, the pre-set deep reinforcement learning network gradually optimizes its network parameters, so that the output action value function value more accurately reflects the long-term cumulative benefit of performing each candidate action.

[0065] S4. Generate cross-connect instructions according to the topology reconstruction strategy and send them to the reconfigurable optical add-drop multiplexer for execution to complete the topology reconstruction of the all-optical network.

[0066] Topology reconfiguration strategy refers to a complete decision-making scheme formed by combining path optimization actions, wavelength scheduling actions, and link switching actions layer by layer in the hierarchical action space. It includes the end-to-end transmission path of the current service request, the wavelength processing method of each intermediate node in the path, and the allocation result of wavelength channels along the entire path. Cross-connection commands are operation commands used to control the wavelength selection switches inside the reconfigurable optical add-drop multiplexer to execute optical layer cross-connection configurations. They transform the abstract topology reconfiguration strategy into hardware control signals that the reconfigurable optical add-drop multiplexer can recognize and execute. By transforming the abstract strategy obtained from hierarchical decision-making into the actual optical path configuration at the physical layer, and updating the entire network resource status table to achieve real-time synchronization of network operating status, a closed loop of topology reconfiguration from decision-making to execution is completed.

[0067] In some implementations, the optimal transmission path output by the path optimization action layer is extracted from the topology reconfiguration strategy and arranged sequentially from the source node to the destination node. This determines all reconfigurable optical add-drop multiplexer (OLD) nodes requiring cross-connect configuration and their execution order. For each OLD node in the optimal transmission path, the corresponding wavelength scheduling action (including wavelength scheduling type, wavelength number, ingress port number, egress port number, and fiber link number) is read from the topology reconfiguration strategy. The actual action to be performed by the node is determined based on its position in the path: if the node is the source node of the current service request, a wavelength addition action is performed, inputting local service traffic into the all-optical network through the addition port; if the node is the destination node of the current service request, a wavelength discard action is performed, outputting signals from the ingress port to the local network through the discard port; if the node is an intermediate node, only a wavelength pass-through action is performed, forwarding signals from the ingress port to the next node from the corresponding egress port. For each node requiring cross-connect configuration, the node identifier, cross-connect action type, wavelength number, ingress port number, and egress port number are encapsulated into a standard-format cross-connect instruction.

[0068] For example, when the wavelength scheduling action of an intermediate node is wavelength pass-through, wavelength is wavelength channel 5, inlet port number is P1, and outlet port number is P2, the generated cross-connect instruction content is "Node N, pass-through, wavelength 5, inlet port P1, outlet port P2"; when the wavelength scheduling action of the source node is wavelength addition, wavelength is wavelength channel 3, add port number is Add1, and outlet port number is P3, the generated cross-connect instruction content is "Node S, add, wavelength 3, add port Add1, outlet port P3".

[0069] Once all cross-connect instructions are generated, they are sequentially sent to the wavelength selection switch controllers of each reconfigurable optical add-drop multiplexer (OCD). Upon receiving the cross-connect instructions, each OCD's wavelength selection switch controller parses the instructions and drives the corresponding microelectromechanical system (MEMS) or liquid crystal switch array to switch the optical signal at the specified input port to the specified output port according to the specified wavelength, thus completing the physical cross-connection of the optical layer.

[0070] During the successful execution of cross-connection commands node by node, the entire network resource status table needs to be updated synchronously. When a cross-connection command for a node is successfully executed, the status of the wavelength channels involved by that node in the corresponding port direction is updated to "occupied." Once the cross-connection commands for all nodes in the optimal transmission path have been successfully executed, the status of the wavelength channels allocated on each fiber optic link in the optimal transmission path is updated to "occupied." This update operation of the entire network resource status table provides an accurate resource occupancy benchmark for subsequent service request decisions, ensuring the real-time accuracy of information such as wavelength occupancy status and port availability during subsequent decision-making processes.

[0071] Based on the above technical solutions, by constructing a multi-dimensional state vector that integrates node operating parameters, link transmission parameters, service request parameters, and overall network performance parameters, a comprehensive and accurate perception of the operating status of the all-optical network is achieved, effectively solving the problem of missing decision-making information caused by the one-sided state representation of traditional methods. By designing a three-layer hierarchical action space that includes path optimization, wavelength scheduling, and link switching, the multi-dimensional control capabilities of the reconfigurable optical add-drop multiplexer are deeply adapted to the decision-making process, significantly reducing the complexity of the action space and improving the flexibility of topology reconfiguration. In the path optimization layer, a multi-level judgment factor mechanism based on the resource status of intermediate nodes is introduced to dynamically consider the accumulated consumption of preceding services and the resolution of concurrent request conflicts, achieving accurate selection of the optimal transmission path. In the wavelength scheduling layer, the physical characteristics of the optical layer are transformed into pass-through action limit values ​​and embedded. Action value function correction enables the pre-defined deep reinforcement learning network to balance transmission distance, wavelength blocking risk, and signal quality attenuation, achieving precise scheduling of wavelength resources within nodes. At the link switching layer, link-by-link verification and partial rollback mechanisms ensure conflict-free allocation and reliable configuration of wavelength channels along the entire path. Simultaneously, a multi-dimensional reward function combining global network performance evaluation and local action cost evaluation is constructed to overcome the problem of traditional reward functions' delayed response to physical characteristic fluctuations. The synergistic effect of these mechanisms enables all-optical network topology reconstruction to accurately match dynamically changing service demands and physical transmission constraints. While improving wavelength resource utilization efficiency and network reconstruction flexibility, it effectively reduces network blocking rate and average transmission loss, achieving efficient and accurate reconstruction of all-optical network topology in dynamic traffic scenarios.

Claims

1. A method for topology reconstruction of an all-optical network based on reinforcement learning, characterized in that, include: Obtain the connection relationships between reconfigurable optical add-drop multiplexers and the link lengths of optical fiber links in an all-optical network, and construct the physical topology of the all-optical network; Obtain network operation status information of all-optical networks and construct multi-dimensional state vectors; Based on the physical topology of the all-optical network, the network operation status information, and the multi-dimensional state vector, decisions are made layer by layer in the hierarchical action space to determine the path optimization action, wavelength scheduling action, and link switching action corresponding to the optimal transmission path of the current service request, and to generate a topology reconstruction strategy. The cross-connection command is generated according to the topology reconstruction strategy and sent to the reconfigurable optical add-drop multiplexer for execution, thereby completing the topology reconstruction of the all-optical network.

2. The all-optical network topology reconstruction method based on reinforcement learning according to claim 1, characterized in that, The network operation status information includes node operation parameters of the reconfigurable optical add-drop multiplexer, transmission parameters of the optical fiber link, service request parameters, and overall network performance parameters. Obtain network operation status information of the all-optical network and construct a multi-dimensional state vector, including: The port occupancy rate, number of wavelengths added, number of wavelengths dropped, optical power loss, and electrical layer processing margin of the reconfigurable optical add-drop multiplexer are obtained to obtain the node operating parameters; The transmission parameters are obtained by acquiring the flow load, wavelength occupancy, transmission loss, and crosstalk noise of the optical fiber link. Obtain the traffic demand, source node and destination node pair distribution, and number of incomplete connection requests for the current business request to obtain the business request parameters; The network-wide wavelength utilization, network-wide blocking rate, and average transmission loss are obtained to acquire the network-wide performance parameters. Based on the distribution of source and destination nodes in the service request parameters, determine the local concentration of traffic, the concentration of head traffic, and the dispersion of traffic. The multidimensional state vector is constructed by fusing the node operating parameters, the transmission parameters, the traffic demand and number of incomplete connection requests in the service request parameters, the local traffic concentration, the header traffic concentration, the traffic dispersion, and the overall network performance parameters.

3. The all-optical network topology reconstruction method based on reinforcement learning according to claim 2, characterized in that, Based on the all-optical network physical topology, the network operating status information, and the multi-dimensional state vector, decisions are made layer by layer in the hierarchical action space to determine the path optimization action, wavelength scheduling action, and link switching action corresponding to the optimal transmission path of the current service request, including: Based on the physical topology of the all-optical network, multiple candidate paths are generated between the source node and the destination node pair; Based on the number of intermediate nodes, port occupancy, optical power loss, and electrical layer processing margin in each candidate path, the overall path cost of each candidate path is determined, and the candidate path with the minimum overall path cost is determined as the optimal transmission path. The path optimization action determines the optimal transmission path based on the physical topology of the all-optical network and the node operating parameters. The multidimensional state vector is input into a preset deep reinforcement learning network to determine the wavelength scheduling action of the intermediate node of the optimal transmission path. Based on the wavelength channel occupancy status of the optical fiber links in the optimal transmission path and the wavelength scheduling action, the wavelength channel allocation result is determined through a link-by-link verification and rollback mechanism, and the link switching action is generated.

4. The all-optical network topology reconstruction method based on reinforcement learning according to claim 3, characterized in that, If there are preceding business requests before the current business request, the comprehensive path cost for each candidate path is determined, which also includes: A first correction coefficient is determined based on the resource occupancy status of intermediate nodes in each candidate path according to the preceding service request; The path composite cost is adjusted according to the first adjustment coefficient, so as to determine the optimal transmission path based on the adjusted path composite cost.

5. The all-optical network topology reconstruction method based on reinforcement learning according to claim 3, characterized in that, If the current business request includes multiple business requests with the same start time, the comprehensive path cost for each candidate path is determined, which also includes: The second correction factor is determined based on the number of common intermediate nodes among the candidate paths of multiple current business requests and the traffic requirements of each business request. The path composite cost is adjusted according to the second correction coefficient, and the optimal transmission path is determined based on the adjusted path composite cost.

6. The all-optical network topology reconstruction method based on reinforcement learning according to claim 3, characterized in that, The multidimensional state vector is input into a preset deep reinforcement learning network to determine the wavelength scheduling action of the intermediate node of the optimal transmission path, including: Based on the wavelength occupancy status, port availability status, and link failure status of each node in the optimal transmission path, a set of optional actions is generated. The multidimensional state vector is input into the preset deep reinforcement learning network to calculate the action value function value of each candidate action in the selectable action set, and the wavelength scheduling action of the intermediate node is determined from the selectable action set based on the action value function value.

7. The all-optical network topology reconstruction method based on reinforcement learning according to claim 6, characterized in that, Based on the wavelength channel occupancy status of the fiber optic links in the optimal transmission path and the wavelength scheduling action, the wavelength channel allocation result is determined through a link-by-link verification and rollback mechanism, and the link switching action is generated, including: Mark the occupancy status of the wavelength channels in the optical fiber link in the optimal transmission path, and determine the set of common wavelength channels that are idle on the optical fiber link. If the set of public wavelength channels is not empty, select a wavelength channel from the set of public wavelength channels as the allocated wavelength channel, and verify the port availability corresponding to the wavelength scheduling action node by node on the allocated wavelength channel. Then, allocate the verified allocated wavelength channel to the optical fiber link in the optimal transmission path. If the set of public wavelength channels is empty, or the port availability verification fails, the optimal candidate path will be reselected.

8. The all-optical network topology reconstruction method based on reinforcement learning according to claim 3, characterized in that, Also includes: The global network performance evaluation value is determined based on the overall network wavelength utilization, the overall network blocking rate, and the average transmission loss. Based on the action type of the wavelength scheduling action, combined with the link length, wavelength occupancy, and cumulative transmission loss of the intermediate nodes of the optimal transmission path, the local action cost evaluation value is determined. Construct a multi-dimensional reward function that includes the global network performance evaluation value and the local action cost evaluation value, and update the preset deep reinforcement learning network according to the multi-dimensional reward function.

9. The all-optical network topology reconstruction method based on reinforcement learning according to claim 2, characterized in that, Based on the distribution of source and destination nodes in the service request parameters, determine the local concentration of traffic, the concentration of header traffic, and the dispersion of traffic, including: The local concentration of traffic is determined by the ratio of the maximum number of requests for the source node and destination node pair to the total number of requests. The headline traffic concentration is determined by the ratio of the sum of the number of requests from the source node and destination node pairs that are ranked from largest to smallest to the total number of requests. The traffic dispersion is determined based on the probability distribution of the source and destination node pairs.

10. A reinforcement learning-based all-optical network topology reconstruction system, characterized in that, include: The topology building module is used to obtain the connection relationships between reconfigurable optical add-drop multiplexers in the all-optical network and the link length of the optical fiber links, and to build the physical topology of the all-optical network. The state awareness module is used to acquire network operation state information of the all-optical network and construct a multi-dimensional state vector; The hierarchical decision module is used to make decisions layer by layer in the hierarchical action space based on the physical topology of the all-optical network, the network operation status information, and the multi-dimensional state vector, to determine the path optimization action, wavelength scheduling action, and link switching action corresponding to the optimal transmission path of the current service request, and to generate a topology reconstruction strategy. The topology reconfiguration module is used to generate cross-connect instructions according to the topology reconfiguration strategy and send them to the reconfigurable optical add-drop multiplexer for execution, thereby completing the topology reconfiguration of the all-optical network.

Images