A photon pulse reinforcement learning intelligent network routing method and system

Abstract

The application discloses a kind of photonic pulse reinforcement learning intelligent network routing method and system. Through the real-time collection of network state information by OpenFlow switch of SDN data plane, and after the normalization processing of host identification and path hop count therein, the environment state vector is constructed in combination with path load characteristics;Intelligent routing decision is executed using pulse reinforcement learning engine, and the core network realizes optimal path selection through pulse coding, pulse neural network layer inference and pulse decoding, wherein the linear calculation part is completed by MZI interferometer for high-speed low-power operation, and the nonlinear pulse activation simulates LIF neuron dynamic response through DFB-SA laser array;Finally, the optimal routing decision is converted into flow table rule and issued to the switch for execution. The application deeply integrates photonic pulse neural network with SDN architecture and PPO algorithm, significantly improves the throughput, load balancing capability of the routing system, and reduces the packet loss rate and transmission delay.

Description

Technical Field

[0001] This invention belongs to the field of communication network and artificial intelligence technology, specifically relating to a photon pulse reinforcement learning intelligent network routing method and system. Background Technology

[0002] With the accelerated pace of global digital transformation, emerging information technologies such as cloud computing, big data, artificial intelligence, and the Internet of Things are profoundly reshaping social production and lifestyles. The explosion of massive data flows and the surge in real-time computing power demands place stringent requirements on the performance, efficiency, and intelligence of underlying network infrastructure. Data center networks, emerging computing networks, and future-oriented 6G communication networks, as the core support of the digital economy, directly impact a nation's core competitiveness in the digital age and the quality of social informatization development. In these large-scale, topologically complex, and dynamically changing network environments, intelligent routing technology, as the "central nervous system" of the network, undertakes the crucial mission of optimizing resource allocation, ensuring service quality, and improving overall network efficiency, and has become a cutting-edge focus in the field of information and communication technology.

[0003] Currently, traditional routing algorithms, represented by Dijkstra's algorithm based on the shortest path and its derivatives in protocols such as OSPF and IS-IS, constitute the cornerstone of routing on the modern Internet and have played an irreplaceable role over the past few decades. However, these algorithms are essentially static or quasi-static deterministic decision-making mechanisms, relying on local or fixed metrics (such as hop count and bandwidth) for path selection. When faced with the dynamic, bursty, and varied traffic characteristics of modern networks, especially data centers and computing power networks, they exhibit significant inherent limitations: First, they lack real-time perception and historical learning capabilities of the global network state, making them "short-sighted" in their decision-making and unable to cope with complex global optimization problems. Second, the fixed rule-based driving characteristics lead to rigid strategies, making them unable to adaptively learn and cope with unknown or complex nonlinear traffic patterns, resulting in slow convergence in network congestion or failure scenarios. Third, their optimization objectives are singular (such as minimizing hop count), making it difficult to achieve effective coordinated trade-offs among multiple competing key performance indicators such as throughput, end-to-end latency, packet loss rate, and load balancing, which can easily lead to problems such as local network congestion and uneven resource utilization.

[0004] To overcome the shortcomings of traditional methods, deep reinforcement learning offers a novel approach to addressing the aforementioned challenges. Reinforcement learning agents can learn complex network dynamics from a high-dimensional state space without explicit programming through continuous interaction with the network environment, thereby approximating globally optimal or suboptimal routing policies. Among numerous reinforcement learning algorithms, the Proximal Policy Optimization (PPO) algorithm demonstrates significant application potential in complex routing optimization tasks due to its good training stability, excellent sample efficiency, and strong support for both continuous and discrete action spaces. However, traditional deep reinforcement learning models based on artificial neural networks face severe engineering and physical bottlenecks in practical deployment: on the one hand, artificial neural network models involve a large number of intensive floating-point matrix operations, resulting in high computational energy consumption and long decision latency on general-purpose processors, contradicting the stringent requirements of low latency and high response in network control planes, especially in future edge computing scenarios; on the other hand, the multiple reinforcement learning agents required for large-scale network deployment will generate considerable overall energy consumption, increasing operating costs and limiting its widespread application in energy-constrained scenarios or those with a strong demand for "green computing."

[0005] Against the backdrop of traditional electronic computing architectures being constrained by the "memory wall" and "power wall," spiking neural networks (SNNs), inspired by the information processing methods of biological nervous systems, and photonic computing technology using photons as information carriers and based on SNNs, offer revolutionary hope for overcoming these performance and energy efficiency bottlenecks. SNNs mimic the working mechanism of the brain, using event-driven pulse sequences for information encoding and transmission. Neurons fire pulses only when the membrane potential reaches a threshold. This sparse and asynchronous computing characteristic, compared to the continuous activation mode of artificial neural networks, theoretically offers lower computational energy consumption and faster response speed. Photonic computing leverages the physical advantages of light, such as high-speed propagation, ultra-high bandwidth, natural parallelism, low crosstalk, and low power consumption, providing a near-ideal hardware platform for spiking neural networks. This makes it possible to achieve nanosecond or even picosecond-level inference latency and ultra-low energy consumption per operation at the milliwatt or even microwatt level at the hardware level for intelligent decision-making. By combining the computational model of spiking neural networks with the physical advantages of photons, photonic spiking neural networks are becoming an important development direction leading the next generation of high-performance, low-power artificial intelligence computing architectures, and have already shown initial success in tasks such as image classification, speech recognition, and high-speed time-series signal processing (such as gene sequence analysis).

[0006] In terms of hardware implementation, the Optical Brain-Inspired Computing Team at Xi'an University of Electronic Science and Technology independently developed a distributed feedback (DFB-SA) semiconductor laser chip with a saturable absorption region as an optical pulse neuron. This successfully simulated neuron-like responses such as excitation response, threshold, accumulation, and refractory period. Based on a traditional InP-based laser technology platform, this photonic pulse neuron features integration, low power consumption, high speed, and easy tuning, making it suitable for applications requiring high bandwidth, high speed, and low latency. Researchers at Beijing University of Posts and Telecommunications proposed a reinforcement learning model (ORL) based on photonic neural networks and applied it to reinforcement learning environments in both discrete and continuous data spaces. Experiments showed that the model could stably find the shortest path after 200 games in a two-dimensional grid world, and after 300 games in a three-dimensional grid world, the number of steps per game decreased from 1000 to approximately 20, validating the effective learning ability and environmental adaptability of photonic neural networks in reinforcement learning tasks. In addition, Jin Xianmin's research group at Shanghai Jiao Tong University developed a hybrid architecture programmable photonic chip (HyArch PIC) and integrated it with a high-speed FPGA and an electric driver to form a highly integrated optoelectronic computing board. This chip can perform high-precision calculations of up to 15-dimensional optical dot product operations, ensuring the effective execution of complex reinforcement learning algorithms. Furthermore, the introduced similarity reward function (SRF) achieved an exponential speedup compared to the constant reward function (CRF) in cliff walking benchmark tests.

[0007] While the combination of photonic spiking neural networks and reinforcement learning has made progress in some areas, research exploring the integration of this emerging technology with the network control plane, especially the critical task of intelligent routing, remains largely unexplored. Current technologies fail to fully leverage the low latency and low power consumption advantages of photonic computing and the event-driven characteristics of spiking neural networks to build efficient intelligent routing systems suitable for SDN architectures. This makes it difficult to meet the urgent demands of future large-scale data centers, computing networks, and 6G networks for low latency, low power consumption, and highly intelligent routing. In other words, current technologies suffer from rigid traditional routing strategies, high energy consumption and latency in electronically-based reinforcement learning, and insufficient integration of photonic computing with SDN routing scenarios. Summary of the Invention

[0008] To address the aforementioned problems in the existing technology, this invention provides a photon pulse reinforcement learning intelligent network routing method and system. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a routing method for photon pulse reinforcement learning intelligent networks, comprising: Obtain the network state information at the current moment, and construct the environment state vector at the current moment based on the network state information at the current moment; The current environmental state vector is modulated onto a multi-channel light source to obtain an optical signal carrying the current environmental state vector. An explicit MZI network with the weight matrix of the linear layer loaded with a pre-trained spiking reinforcement learning engine is used to perform linear calculations on the input optical signal to obtain an interference optical signal. The interference optical signal is then conditioned and converted to obtain a linear electrical signal. The pre-trained spiking reinforcement learning engine includes a pre-trained spiking Actor network, which contains an input encoding layer, N hidden layers, and an output decoding layer. Each hidden layer adopts a spiking neural network structure. The pre-trained spiking reinforcement learning engine is trained using the pruning policy gradient optimization and generalized advantage estimation method of the PPO algorithm. N is a positive integer greater than or equal to 1. Using a DFB-SA laser array to simulate neurons in the hidden layer of the pre-trained pulse actor network that employ the LIF neuron model, the input linear electrical signal is subjected to membrane potential accumulation, threshold judgment, and pulse firing to obtain a pulse sequence. Based on the pulse sequence output by the DFB-SA laser array of neurons using the LIF neuron model in the last hidden layer of the pre-trained pulsed actor network, an optimal routing path is generated to generate SDN routing control instructions based on the optimal routing path, driving the data plane to perform traffic forwarding tasks.

[0009] This invention also provides a photon pulse reinforcement learning intelligent network routing system for implementing the steps of the above-described photon pulse reinforcement learning intelligent network routing method, the system comprising: The SDN controller is used to acquire the network status information at the current moment and construct the environment state vector at the current moment based on the network status information at the current moment. A signal modulator is used to modulate the current environmental state vector onto a multi-channel light source to obtain an optical signal carrying the current environmental state vector. An explicit MZI network is loaded with the weight matrix of a linear layer of a pre-trained pulse reinforcement learning engine, which is used to perform linear calculations on the injected optical signal to obtain the interference optical signal. The signal conditioning and conversion module is used to condition and convert the interference optical signal output by the explicit MZI network to obtain a linear electrical signal; The DFB-SA laser array contains multiple DFB-SA lasers. Each DFB-SA laser is used to simulate a neuron in the hidden layer of the pre-trained pulse Actor network using the LIF neuron model. It is used to accumulate membrane potential, determine thresholds, and fire pulses on the input linear electrical signal to obtain a pulse sequence. The SDN controller is further configured to generate an optimal routing path based on the pulse sequence output by the DFB-SA laser array of neurons using the LIF neuron model in the last hidden layer of the pre-trained pulsed actor network, and to generate SDN routing control instructions based on the optimal routing path to drive the data plane to perform traffic forwarding tasks.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: 1) This invention collects multi-dimensional network status information through OpenFlow switches in the SDN data plane. After standardization, it combines host encoding and path features to construct an environment status vector. This can transform heterogeneous, high-dimensional network operation data into structured low-dimensional vectors, which not only retains the key information required for routing decisions (host identifier, path load, hop count features, etc.) but also reduces the complexity of subsequent calculations. This provides an accurate and efficient input foundation for intelligent decision-making. At the same time, the fixed vector dimension ensures the stability of algorithm training and inference.

[0011] 2) This invention utilizes the sparse computation characteristics of event-driven spiking neural networks. Neurons only fire pulses when the membrane potential reaches a threshold. Compared with the intensive computation of traditional artificial neural networks, this significantly reduces the energy consumption of routing decisions. Furthermore, the temporal integration capability of LIF neurons can better adapt to the dynamic changes in network traffic, thereby improving the environmental adaptability of the strategy.

[0012] 3) This invention maps the weights of the pulse Actor network to an MZI interferometer to complete linear calculations, giving full play to the advantages of high parallelism, high-speed propagation and low power consumption of photonic computing, realizing physical acceleration of interlayer linear transformations. Optical domain interferometric calculations can complete large-scale matrix operations in nanoseconds, breaking through the "speed-power consumption" bottleneck of traditional electronic computing, and providing core hardware support for low latency routing decisions.

[0013] 4) This invention injects linear electrical signals into a DFB-SA laser array to achieve nonlinear pulse activation. It can utilize the physical characteristics of the DFB-SA laser to simulate the threshold triggering and pulse firing functions of biological neurons, and complete the efficient signal conversion of "optical-electrical-optical". It maintains the advantages of low latency and low power consumption throughout the entire link, and realizes the nonlinear inference of the spiking neural network. It is a key technology link for the application of photonic neuromorphic computing in routing scenarios.

[0014] 5) This invention optimizes routing strategies through the pruning strategy and generalized advantage estimation mechanism of the PPO algorithm, and combines it with an SDN controller to achieve flow table distribution and closed-loop control. It can deeply integrate the autonomous optimization capability of reinforcement learning with the centralized control advantage of SDN, enabling the routing strategy to adapt to changes in network topology and traffic pattern migration. It achieves collaborative optimization among multiple objectives such as throughput improvement, latency reduction, packet loss rate control and load balancing. Compared with the traditional Dijkstra algorithm, its performance advantage is particularly significant in high-load scenarios, providing an efficient, intelligent and green routing solution for data centers, computing networks and other scenarios.

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the photon pulse reinforcement learning intelligent network routing method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the network structure of the pulse Actor network provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the Spiking-PPO training framework provided in an embodiment of the present invention; Figure 4 This is a diagram showing the curves of reward and loss values ​​changing with the number of training rounds, and a performance comparison between the PPO algorithm and the Dijkstra algorithm, provided in an embodiment of the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0018] Figure 1 This is a flowchart illustrating a photon pulse reinforcement learning intelligent network routing method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes: S101. Obtain the network state information at the current moment, and construct the environment state vector at the current moment based on the network state information at the current moment.

[0019] S102. Modulate the current environmental state vector onto a multi-channel light source to obtain an optical signal carrying the current environmental state vector.

[0020] For example, using a multi-channel light source with wavelengths spaced 0.5 nm apart as the input signal source, and employing a Mach-Zehnder modulator (MZM) to modulate the light source with pulse signals, an optical signal carrying the current environmental state vector is obtained. For instance, the optical signal wavelength can be selected from the 1550 nm communication band.

[0021] S103. Using an explicit MZI network with the weight matrix of the linear layer loaded with a pre-trained spiking reinforcement learning engine, linear calculations are performed on the input optical signal to obtain the interference optical signal. The interference optical signal is then conditioned and converted to obtain a linear electrical signal. The pre-trained spiking reinforcement learning engine contains a pre-trained spiking Actor network, which includes an input encoding layer, N hidden layers, and an output decoding layer. Each hidden layer adopts the structure of a spiking neural network. The pre-trained spiking reinforcement learning engine is trained using the pruning policy gradient optimization and generalized advantage estimation method of the PPO algorithm. N is a positive integer greater than or equal to 1.

[0022] Here, the pre-trained spiking reinforcement learning engine is a pre-trained spiking Actor network. The spiking Actor network consists of an input encoding layer, N hidden layers, and an output decoding layer. Each hidden layer is an SNN network, meaning the network architecture of each hidden layer is identical to that of an SNN network. Therefore, each hidden layer consists of a fully connected layer (i.e., a linear layer) + a spiking layer (i.e., a Leaky Integrate and Fire (LIF) layer). Each neuron in the spiking layer of each hidden layer is a neuron using the LIF neuron model, and each hidden layer possesses the characteristics of an SNN network. The spiking Actor network proposed in this invention achieves backpropagation of the spiking mechanism during training through a differentiable approximation function. Specifically, the SNN network used in this invention is based on the LIF neuron model and achieves information processing by simulating the membrane potential leakage accumulation and spiking triggering mechanism of biological neurons. Its working principle is as follows: For discrete time steps (Right now (Time), defining the membrane potential of a neuron as The input current (or input signal) is The membrane potential leakage coefficient is (0< <1), the pulse firing threshold is Within each time step, the membrane potential update of this neuron follows these rules: ; ; Represents the pulse firing function, when the membrane potential... Exceeding the threshold At this time, the neuron fires a pulse, simultaneously resetting the membrane potential to... (or approximately reset to 0), completing the "discharge" process. In the above formula, the leakage coefficient... The value of determines the memory characteristics of the membrane potential: The smaller the value, the faster the historical information of the membrane potential decays; The larger the value, the better the membrane potential retains historical inputs.

[0023] Due to the pulse firing function Essentially discontinuous and non-differentiable, this invention employs an approximate gradient technique called ZIF (Zero-Inflated Firing) (or other equivalent approximate gradient techniques) to approximate the pulse firing function as a continuous function, allowing backpropagation to occur within the network. Specifically, this is achieved through an approximate activation function. Replacement step The approximate relationship is as follows: ; Simultaneously, gradient clipping is introduced during backpropagation: Alternatively, a smoothing function can be used to effectively calculate the membrane potential gradient, thus ensuring the feasibility of training the spiking neural network. This indicates a cropping operation.

[0024] The pulse Actor network proposed in this invention processes each input environmental state vector as follows: (1) The input environmental state vector is converted into a pulse sequence and passed to the first hidden layer through the input encoding layer. (2) The input pulse sequence is linearly transformed through the linear layer in the first hidden layer to obtain the neuron input current. Then, the pulse sequence is generated according to the input current and passed to the next hidden layer through the membrane potential accumulation and threshold triggering characteristics of the LIF layer in the first hidden layer using the LIF neuron model. The next hidden layer processes the input pulse sequence using the same principle as the first hidden layer until the last hidden layer outputs the pulse sequence to the output decoding layer. Specifically, the LIF layer in each hidden layer is based on the LIF neuron model and uses the leakage integral mechanism to accumulate the input current and the membrane potential at the previous moment to obtain the current membrane potential. Then, the current membrane potential is compared with the firing threshold. If it exceeds the threshold, a pulse is fired and the membrane potential is reset. Otherwise, the membrane potential is maintained. Finally, the obtained binary pulse signal (i.e., pulse sequence) is used as the output of the hidden layer and passed to the next layer. (3) The output decoding layer generates an original output vector based on the input pulse sequence. Each element in the original output vector represents a routing path. Infeasible paths in the original output vector are filtered by an action mask to obtain the output vector after the action mask. The output vector after the action mask is converted into a path selection probability distribution by the Softmax function, and the routing path with the highest probability is selected from the path selection probability distribution for output. The method of filtering infeasible paths in the original output vector by the action mask includes: for each element in the original output vector, determining whether the element position index is less than the number of actual available paths. If yes, the action mask corresponding to the element is set to 1 (valid); otherwise, the action mask corresponding to the element is set to 0 (invalid). Then, the elements in the original output vector whose action mask is 0 are set to a very small negative value (such as -1e9) to obtain the output vector after action masking.

[0025] Specifically, to achieve the extraction of temporal features of SDN network states and the output of routing / scheduling actions, the pulse Actor network designed in this invention adopts a temporally unfolded multilayer perceptron structure. Each time step performs a three-stage operation: "linear weighting—membrane potential integration—threshold firing." For example, the network structure of the pulse Actor network is as follows: Figure 2 As shown, it includes one input layer (i.e., the input encoding layer mentioned above), two hidden layers, and one output layer (i.e., the output decoding layer mentioned above). The specific calculation process of each layer is as follows: Assuming the input state (Right now (environmental state vector at each time step), parameters of each layer and These represent the weights and biases of each layer, ∈{0,1} is the first i Layer neurons at time t (Right now Pulse output at (time) i If ∈{1,2,3}, then: Input layer: Receive state and status Convert to a pulse sequence, convert the pulse sequence Output to the first hidden layer; First hidden layer: First, process the input pulse sequence through a linear layer. Perform linear calculations to obtain the neuron's input current. : Subsequently, membrane potential updates and pulse firing are performed to obtain a pulse sequence. : , Finally, the pulse sequence Output to the second hidden layer; The second hidden layer: first, the input pulse sequence is processed through a linear layer. Perform linear calculations to obtain the neuron's input current. : Subsequently, membrane potential updates and pulse firing are performed to obtain a pulse sequence. : ; Finally, the pulse sequence Output to output layer; Output layer: First, process the input pulse sequence... Perform linear calculations to obtain the original output vector. : Original output vector (The following is represented as) () represents the initial routing path, such as Figure 3 As shown, It is an 8-dimensional vector; then, the original output vector is filtered using an action mask. After identifying infeasible paths, the algorithm transforms them into a path selection probability distribution using the Softmax function, ultimately outputting the SDN routing decision action. (Right now The optimal route path at time (i.e., the action mask). The action mask is a boolean vector used to indicate which route paths (i.e., actions) are valid; for example, the original output vector. The Middle Action mask corresponding to each element The expression is: ,in, Represents the actual number of available paths, when When, it indicates the original output vector The Middle The routing path corresponding to each element exists and is feasible, while when When, it indicates the original output vector The Middle The routing path corresponding to each element does not exist or is infeasible. This is used to obtain the original output vector. After encoding the action mask for each element, the action mask is applied to the original output vector. The output vector after action masking can be obtained. Specifically, for the original output vector The Middle element ,like Corresponding action mask Then the original output vector is retained. The Middle element ,like Corresponding action mask Then the original output vector The Middle element Set as (This is actually a very small negative value (e.g., -1e9)), thus obtaining the output vector after action masking. ,Right now Finally, the output vector after action masking is processed by the Softmax function. This is converted into a path selection probability distribution; specifically... It is converted into path selection probability through the Softmax function. : Then, the routing path with the highest path selection probability is selected as the SDN routing decision action. Output. In this invention, the dynamic membrane potential and temporal superposition mechanism of SNN enable it to capture the dynamic changes of SDN networks and maintain decision stability in scenarios such as topology reconstruction and traffic fluctuations.

[0026] In this invention, the weight matrix of each linear layer of a pre-trained pulse reinforcement learning engine can be loaded into a corresponding explicit MZI network, enabling the explicit MZI network to perform linear calculations for its corresponding linear layer. Thus, the weight matrix can be optically fixed to the explicit MZI chip via voltage control, resulting in multiple explicit MZI networks loaded with different weight matrices. Furthermore, each explicit MZI network performs a different linear calculation function, achieving precise mapping between weights and hardware, and completing the hardware deployment of the weights. Specifically, each MZI network consists of P*P connected MZI units, where the value of P is determined according to actual needs. Each MZI unit represents a weight value, and each MZI unit is a 2×2 basic MZI unit. Each 2×2 basic MZI unit integrates four electro-optic phase shifters, two for the inner arm and two for the outer arm. Each MZI network consists of P×P 2×2 MZI basic units cascaded and cross-connected. For example, each MZI network consists of 4×4 or 16×16 2×2 MZI basic units cascaded and cross-connected. The weight value represented by each MZI unit is calculated as follows: ,in, This indicates the phase shift of the electro-optic phase shifter inside the MZI unit, and, by adjusting... The size of the MZI cell controls the weight value it represents. This indicates the weight value represented by the MZI cell. The imaginary unit is used because, in the weight calculation of the MZI unit, the transformation matrix of the MZI is a matrix in the complex field (since the modulation of optical signals involves complex field operations of phase and amplitude). Therefore, the imaginary unit is required. This describes the phase / amplitude relationship of complex signals. Each explicit MZI network, loaded with a corresponding weight matrix, can perform a linear product calculation between the input and the weight matrix through optical interference, thereby obtaining the interference optical signal.

[0027] S104. Using a DFB-SA laser array with neurons in the hidden layer of the pulse Actor network used for simulation pre-training that employ the LIF neuron model, the input linear electrical signal is subjected to membrane potential accumulation, threshold judgment, and pulse firing to obtain a pulse sequence.

[0028] In this invention, a DFB-SA laser array can be used to simulate all neurons in the pulse layer of a hidden layer in a pre-trained spiking Actor network. The DFB-SA laser array contains multiple DFB-SA lasers, each simulating one neuron, and these lasers are connected according to the connectivity relationships between the simulated neurons. Thus, when the pre-trained spiking Actor network contains Q hidden layers, Q DFB-SA laser arrays can be used to simulate the neurons in the pulse layer of each of these Q hidden layers. Specifically, the q-th DFB-SA laser array is used to simulate all neurons in the pulse layer of the q-th hidden layer, where q is a positive integer and its value ranges from 1 to Q.

[0029] Specifically, when using a DFB-SA laser to simulate a neuron based on a LIF neuron model, the basic firing threshold (i.e., firing threshold) of the neuron simulated by that DFB-SA laser can be set by independently adjusting the gain region current of each DFB-SA laser through the driving circuit. Simultaneously, by adjusting the reverse bias voltage in the saturation region (SA) to regulate the carrier lifetime in the saturation region, the refractory period and dynamic response characteristics of the simulated neuron can be configured to ensure that the neuron-like function of the simulated neuron accurately matches the design requirements of the spiking neural network. That is, by adjusting the current in the gain region and the reverse bias voltage in the saturation region of the DFB-SA laser, and setting the neuron-like state of the DFB-SA laser, the DFB-SA laser can generate a neuron-like response to the input signal, thereby obtaining a pulsed output signal. It should be noted that the specific adjustment method and principle are not limited in this invention. Accordingly, the specific principle for generating a pulse sequence using a DFB-SA laser array of all neurons within any simulated pulse layer is as follows: The corresponding linear electrical signals are injected into each independent DFB-SA laser in the DFB-SA laser array through a high-speed driving circuit array according to the neuron correspondence, thereby driving the corresponding devices in the DFB-SA laser array. This allows each DFB-SA laser to independently process the linear output of a neuron. When the intensity of the input linear electrical signal exceeds the basic excitation threshold of the corresponding DFB-SA laser, the DFB-SA laser generates a narrow pulse light output (i.e., a narrow pulse light signal with a high extinction ratio (typically ≥15dB)) to achieve the nonlinear activation function of the neuron-like signal. Therefore, all the pulse light outputs generated by the DFB-SA laser array constitute the generated pulse sequence.

[0030] S105. Based on the pulse sequence output by the DFB-SA laser array of neurons in the last hidden layer of the pulse Actor network using the LIF neuron model for simulation pre-training, generate an optimal routing path to generate SDN routing control instructions based on the optimal routing path, driving the data plane to perform traffic forwarding tasks.

[0031] In some embodiments, the network state information at the current moment includes: network topology structure information, all data requests at the current moment, the current load of each link in the network topology, and link capacity. Based on this, S101 is implemented through steps S1011~S1013: S1011. For each data request at the current moment, extract the index information of the source host and the destination host from the data request, and after normalizing the index information, generate a host encoding vector composed of the normalized index information of the source host and the normalized index information of the destination host.

[0032] Here, each data request receives a 2-dimensional host-encoded vector. ,in, This represents the normalized index information of the source host. This represents the index information of the target host after normalization.

[0033] For example, , This is the index information for the source host. Index information for the destination host. This represents the number of indexes for the source and destination hosts in the network topology.

[0034] S1012. Based on the network topology structure information, determine each candidate path between the source host and the destination host in the data request. Based on the network topology structure information, the current load of each link in the network topology, and the link capacity, determine the average utilization, maximum utilization, and normalized hop count of each candidate path between the source host and the destination host in the data request, forming a path feature vector corresponding to each candidate path.

[0035] For example, the first... Average utilization of candidate paths Maximum utilization rate and the normalized jump number The calculation formulas are as follows: ; in, E p To represent the first The set of edges of candidate paths, representing the first edge between the source host and the destination host in any data request. A candidate path refers to any candidate path between the source host and the destination host in any given data request. Indicates the first Links in the candidate paths The current load, For link capacity (e.g., 1000Mbps). Indicates the first The actual number of hops for each candidate path.

[0036] For example, Figure 3 This is a schematic diagram of the Spiking-PPO training framework provided in an embodiment of the present invention. When the network structure is... Figure 3 When a fat tree topology with k=4 (containing 16 hosts and 20 switches) is shown, the path length can be sorted, and 8 candidate paths between the source host and the target host can be selected. The average utilization, maximum utilization, and normalized hop count of each candidate path can be calculated one by one, so that each path feature vector yields 8 3-dimensional path feature vectors, resulting in a total of 24-dimensional path features.

[0037] S1013. Concatenate the host encoding vector and path feature vector corresponding to all data requests at the current moment in order to obtain the environment state vector at the current moment.

[0038] For example, when each data request receives a 2-dimensional host encoding vector and eight 3-dimensional path feature vectors, the host encoding vector and path feature vectors can be concatenated sequentially to obtain a 26-dimensional environment state vector for each data request.

[0039] For example, the logic for constructing the environment state vector is defined in the get_state() method of the SDNEnvironment class, and its core code logic is as follows: Topology and traffic initialization: First, construct a fat tree topology with k=4 (containing 16 hosts and 20 switches), and set the link capacity to 1000Mbps; then generate a fixed-mode traffic matrix (default 64 pairs of host traffic, base rate 100Mbps), and sort the traffic pairs by host index to ensure state consistency. Status acquisition process: When calling env.get_state(), first process OpenFlow messages (status requests / responses, flow table cleanup), and obtain real-time link load from the switch; Host encoding calculation: Extract the source / destination host index of the current traffic pair, normalize it through normalization, and generate the first 2-dimensional host encoding vector; Path feature extraction: Obtain candidate paths between source and destination hosts, sort them by path length, take the top 8 paths, calculate the average utilization, maximum utilization, and normalized hop count for each path, and fill in the subsequent 24-dimensional path features. Vector Assembly and Return: Initialize a 26-dimensional zero vector, assign values ​​in the order of "host encoding + 8 path features", and finally output the environment state vector for the PPOAgent's select_action() interface to call. Therefore, the environment state vector can be represented as: ,in, These represent the average utilization, maximum utilization, and normalized hop count of the first candidate path between the source host and the destination host in the data request, respectively. The other parameters are similar.

[0040] Figure 3 This is a schematic diagram of the overall Spiking-PPO training framework provided in an embodiment of the present invention. See also... Figure 3 This invention employs the Proximal Policy Optimization (PPO) algorithm to train a spurious reinforcement learning engine, thereby obtaining a pre-trained spurious reinforcement learning engine to achieve network traffic scheduling and routing optimization. The PPO algorithm, by introducing a pruning objective function and importance sampling techniques, avoids overexploration while ensuring policy update stability. Compared to traditional reinforcement learning algorithms (such as DQN and DDPG), it is more suitable for the high-dimensional state space and continuous action decision-making requirements of SDN dynamic topologies, and can significantly improve network throughput and reduce transmission latency. For example, the training method for the pre-trained spurious reinforcement learning engine includes steps S1-S10: S1. Initialize the pulsating Actor network, Critic network, and Adam optimizer, and configure the hyperparameters of the PPO algorithm.

[0041] In this invention, the pre-trained spurious reinforcement learning engine is obtained by training an initial spurious reinforcement learning engine, which includes an initial Critic network and an initial spurious Actor network. The hyperparameters of the PPO algorithm include: learning rate and pruning coefficient. For example, during training, the initial exploration rate is set to 1.0, and after each training round, it adaptively decreases by a decay coefficient of 0.99, down to a minimum of 0.05, to balance exploration and utilization. It should be noted that the pulse Actor network in this invention is called a photon pulse Actor network.

[0042] S2. Control the SDN environment to execute routing actions and collect experience data. Each piece of experience data includes the current state, current action, current reward, next state, and round termination flag. For example, at any given time... The corresponding empirical data is ( , , , , , action_mask, ), wherein, is the round termination flag at time . When , it indicates that time is the last step of the round, and there is no subsequent reward. action_mask is action mask of represents the target value calculated based on the generalized advantage estimation method, represents the immediate reward at time , that is, the reward obtained after the SDN environment executes .

[0043] In the present invention, the design of the reward mechanism is adapted to the optimization objectives of SDN, the reward function integrates the core performance indicators of SDN, and a multi-dimensional reward and punishment mechanism is adopted. Specifically, the reward function comprehensively considers multiple optimization objectives such as transmission efficiency, resource utilization and load balancing. Illustratively, has the following expression: ; ; ; ; ; ; ; ; wherein, represents a preset basic reward value, for example, the value is 30.0, so as to provide a stable positive reward baseline for encouraging exploration; represents traffic transmission reward, D represents successfully transmitted traffic, with a unit of Mbps, which is used to encourage high-throughput transmission; P drop represents a packet loss penalty, which is used to punish packet loss, and the penalty intensity increases with the increase of traffic; represents the target traffic volume of transmission, represents a load-related penalty, U net represents network utilization, represents average network utilization, represents maximum network utilization, represents a hop count penalty, This represents the number of hops along the path, which is used to encourage the selection of shorter paths and reduce network resource consumption. This represents long-term load balancing rewards. This represents the long-term load balancing level (ranging from 0 to 1), used to encourage maintaining overall network load balancing. Therefore, The expanded form is: It should be noted that after calculating the reward value using the above formula, the reward value needs to be trimmed to ensure that it falls within the range [-40.0, 100.0] to prevent the reward from exploding.

[0044] S3. Store the obtained experience data in the experience playback buffer.

[0045] It should be noted that during training, after each routing action is completed, the SDN environment will automatically update the link load and regenerate the state vector, storing the experience data such as state, action, and reward into the experience replay buffer to provide input for the subsequent training of the PPO agent.

[0046] Refer to the above Figure 3 The SDN environment includes the control plane where the SDN controller resides and the data plane where switches and hosts reside. The SDN environment collects network status information (e.g., link load, traffic distribution, etc.) through the OpenFlow protocol and receives routing actions output by the PPO agent. According to the received routing path After the corresponding traffic forwarding is executed, an instant reward is returned. With the next state The experience data generated by the PPO agent's interaction with the SDN environment is stored in an experience replay buffer, which uses a first-in-first-out (FIFO) mechanism to retain the latest experience data. Once enough experience data has been accumulated, a batch of samples (e.g., batch size=32) is randomly sampled for network parameter updates.

[0047] S4. Sample batch data from the experience replay pool, calculate the advantage function and target value based on the generalized advantage estimation mechanism, and update the network parameters of the Critic network.

[0048] S5. Update the network parameters of the pulsating Actor network using the PPO pruning strategy and entropy regularization term.

[0049] In combination with the above Figure 3 The calculation process for updating network parameters is as follows: 1) Extended Value Vector: The state value vector output by the Critic network is extended. Value estimation of the last step of the round By piecing them together, we obtain the complete value sequence. Indicates the total number of moves in the rounds; 2) Reverse Iterative Calculation: Due to the last step of the round (i.e. The dominant function value =0, therefore, using the complete value sequence, and according to the time series difference error formula and the GAE reverse recursive formula, from the last step of the round (i.e. Starting from this point, the dominant function value at each time step is calculated sequentially using the recursive formula. The formula for timing difference error is: , For a moment Instant rewards; This is a long-term reward discount factor, with a value of 0.99. It is a moment The state value vector, The set of trainable parameters is The Critic network The value estimate (i.e., the valuation of) (The value output by the Critic network after input). For the Critic network to determine the next state Value estimation; The round end flag ( This indicates that the current move is the last step of the round, and there are no further rewards); the GAE reverse recursive formula is: ,in, This is the GAE tradeoff factor, with a value of 0.96.

[0050] 3) Standardization: The batch dominance function values ​​are standardized by subtracting the mean and dividing by the standard deviation to obtain the standardized dominance function values ​​used for updating the network parameters of the impulse Actor network. Standardized dominance function value The expression is: ,in, Indicates time The dominant function value, The mean of the dominant function values ​​for the batch. The standard deviation of the batch advantage function values ​​is used to improve training stability.

[0051] 4) Network parameter update: 4a) Updating the network parameters of the Critic network: The mean squared error (MSE) is used to minimize the state value estimation error. The expression for the loss function of the Critic network is: ,in, Indicates batch size and status. Indicates time Environment state vector, action Indicates the state The following is the routing path output by the pulse Actor network. It is a moment The state value vector, The set of trainable parameters is The Critic network The value estimate (i.e., the valuation of) The value output by the Critic network after inputting into it. The target value calculated based on the generalized advantage estimation method is expressed as: .

[0052] 4b) Update of the network parameters of the pulsating Actor network: In each iteration, the network parameters of the pulsating Actor network are updated through 15 rounds of batch sampling (i.e., each batch of samples updates 15 times). Each batch contains 32 samples. Gradient clipping (maximum gradient norm 0.5) is used to avoid gradient explosion, and an entropy regularization term (entropy coefficient 0.05) is introduced to encourage policy exploration. The loss function of the spiking Actor network is expressed as follows: ; in, Indicates time The environment state vector, time The state value vector, The set of trainable parameters is The Critic network for state The value estimate, The set of trainable parameters is The pulse Actor network in state Select action The probability, This indicates that the pulse Actor network was in state before the parameter update. Select action The probability of; Indicates time The dominant function value, Indicates time The dominant function value, Indicates time The standardized value of the dominance function; This represents the cutoff factor for PPO. Represents the coefficient of the entropy regularization term. Entropy represents the distribution of actions; This represents the target value calculated based on the generalized advantage estimation method; Indicates time Instant rewards This represents the long-term reward discount factor. Indicates the end of the round. Indicates the GAE trade-off factor. This indicates a cropping operation.

[0053] S6. Limit the magnitude of network parameter updates by using gradient pruning to avoid training oscillations.

[0054] S7. Repeat steps S2 to S6 until the network performance converges to obtain the trained spiking Actor network. Use the trained spiking Actor network as a pre-trained spiking reinforcement learning engine.

[0055] For example, when the obtained loss value reaches the preset loss value, the network performance can be considered converged, and training can be stopped. It should be noted that after each round of network parameter updates, the optimal model parameters (including Actor, Critic network weights and optimizer state) are saved. After training is completed, the model can be loaded using the load_model function and directly used for SDN routing decision inference.

[0056] In some embodiments, the training process of the spiking reinforcement learning engine can be summarized as steps a~k: a. Initialize the spiking Actor network and Critic network, configure the relevant hyperparameters of PPO, and initialize the optimizer and experience replay buffer; b. Set the initial value of the exploration rate, prepare the reward record cache, and start the training loop; c. For each training round, first reset the network environment to obtain the initial state, and initialize the reward of the current round to 0; d. Execute the following operations in a loop within the round: first obtain the dimension information of the action, and end the current round if the dimension is invalid; based on the current network state, action dimension, and exploration rate, select the corresponding routing action and record the probability information of the action; execute the action in the environment to obtain the new network state, immediate reward, and whether the round has ended; e. Store the current state, executed action, action probability, obtained reward, new state, and round end flag into the experience replay buffer, accumulate the reward of the current round, and update the state to the new state until the round ends; f. Extract a batch of data from the experience replay buffer, calculate the value corresponding to the state through the Critic network, and then combine it with the generalized advantage estimation method to obtain the advantage function and target value; g. The data is shuffled and divided into several segments. Each segment is processed sequentially: h. The value of the current state is calculated using the Critic network, and the probability of the action is calculated using the Impulse Actor network; i. The pruning loss is calculated based on the PPO pruning rule, and then the total loss value is obtained by combining the value loss and the entropy regularization term; j. The parameters of the Impulse Actor network and the Critic network are updated through backpropagation, and the gradient is constrained; k. The exploration rate is reduced according to the set decay rule (ensuring that it is not lower than the minimum exploration rate); j. Steps c to j are repeated until the set maximum number of training rounds is reached or the network performance stabilizes and converges, at which point training ends.

[0057] To verify the performance of this invention, the initial spurious reinforcement learning engine was iteratively trained for 3000 rounds based on the PPO algorithm under the above process, and compared with the traditional Dijkstra algorithm. The changes in reward and loss values ​​with the number of training rounds, as well as the performance comparison between the PPO algorithm and the Dijkstra algorithm (including throughput, packet loss rate, average latency, and load balancing), are shown in the following results. Figure 4 As shown. The definitions of throughput, packet loss rate, average latency, and load balancing are as follows: (1) Throughput: refers to the amount of data successfully transmitted in the network per unit time, reflecting the overall data carrying capacity of the SDN network, and is affected by link bandwidth, device forwarding capacity and traffic scheduling strategy.

[0058] ; in, ap For the statistical period Δt Total amount of data successfully transferred (bits), Δ t The statistical period is measured in seconds (s).

[0059] (2) Packet Loss Rate: refers to the proportion of data packets that fail to be successfully transmitted to the destination node within the statistical period to the total number of data packets sent. It reflects the reliability of the SDN network and is usually caused by link overload, insufficient resources or equipment failure.

[0060] ; in, dp The total number of data packets successfully received. tp The total number of data packets sent. tm This refers to the network data transmission rate (Mbps).

[0061] (3) Average delay refers to the average time it takes for a data packet to be sent from the source node to the destination node in the network. It includes four parts: transmission delay, processing delay, queuing delay and propagation delay, and directly reflects the real-time performance of the SDN network.

[0062] ; in, pd The end-to-end delay for each transmission path, d For the number of successful transmissions for each path, Indicates the first The end-to-end delay of the transmission path corresponding to each data packet Indicates the first Successful transmission count of the transmission path corresponding to each data packet tt The total transmission volume across all paths. This represents the total number of data packets within the statistical period.

[0063] (4) Load Balance: This measures the evenness of network traffic distribution across links. The calculation steps for load balance are as follows: First, calculate the average load. L avg and load variance σ ², the formula is as follows: ; in, n For the quantity of resources, l For each resource, calculate the actual load, such as bandwidth utilization. Then calculate the maximum possible variance σ. 2 max : ; in,L max Maximum capacity per link (Mbps) L avg_max For ideal average load, L avg_max = L max / n Finally, the final load balancing score is obtained: .

[0064] As mentioned above Figure 4 As shown, with the increase in the number of interactions with the SDN environment and the continuous optimization of network parameters, the method of this invention enables SDN to achieve efficient routing decisions, load balancing, and QoS guarantees in complex scenarios such as dynamic traffic and topology changes. Compared with the traditional Dijkstra algorithm, the method of this invention improves throughput by approximately 50% and reduces transmission latency by approximately 75%.

[0065] In some embodiments, when the weight matrix of the fully connected layer in the first hidden layer of a pre-trained spiking Actor network is loaded into an explicit MZI network, then S103 above includes steps S1031 to S1032: S1031. The light signal carrying the current environmental state vector is input into an explicit MZI network loaded with the weight matrix of the fully connected layer in the first hidden layer of the pre-trained Actor network. The explicit MZI network then performs a linear product calculation of the current environmental state vector and the weight matrix through optical interference to obtain the interference light signal.

[0066] S1032. The interference light signal is sequentially processed through photoelectric detection, current summation and transimpedance amplification to complete photoelectric conversion and signal conditioning, resulting in a linear electrical signal.

[0067] Specifically, an optical signal carrying the current environmental state vector is injected through a single-mode fiber into the input port of an explicit MZI network loaded with the weight matrix of the fully connected layer in the first hidden layer of a pre-trained Actor network. After entering the explicit MZI network, the optical signal undergoes optical interference in each MZI unit. The interference effect enables parallel multiplication of the state vector and the weight matrix, yielding an interference optical signal. Subsequently, a balanced photodetector (BPD) receives the interference optical signal and converts it into a current signal. The resulting current signals are then summed to perform vector accumulation. Finally, a transimpedance amplifier (TIA) performs current-to-voltage conversion and signal amplification, ultimately completing the process. The matrix-vector multiplication is used to obtain and output a linear electrical signal. This process fully leverages the parallelism of photonic computing, achieving low-latency, high-energy-efficiency linear computation, which is highly suitable for the real-time requirements of SDN routing decisions.

[0068] In some embodiments, the above-mentioned S105 is implemented through steps S1051 to S1054: S1051. The pulse sequence output by the DFB-SA laser array of the neurons in the last hidden layer of the pulse Actor network used for simulation pre-training, which adopts the LIF neuron model, is averaged and normalized in the time dimension to obtain the original path probability vector.

[0069] S1052. Filter infeasible paths in the original path probability vector based on action mask to obtain the path probability vector after action masking.

[0070] Specifically, the logarithmic probability of infeasible paths (e.g., routing paths with link failures, insufficient bandwidth, etc.) in the original path probability vector is set to a minimum value to filter out invalid paths.

[0071] S1053 uses the Softmax function to map the path probability vector after action masking to the selection probability distribution of each candidate route path, and selects the route path with the highest probability as the optimal route path.

[0072] S1054 generates OpenFlow flow table entries based on the optimal routing path and sends the OpenFlow flow table entries to the data plane switch to drive the switch to forward traffic along the specified path.

[0073] Specifically, the discrete pulse sequence output by the DFB-SA laser array is converted into an electrical signal by a photodetector. Pulse counting and averaging are then performed through time window integration to obtain continuous values ​​corresponding to each candidate path, forming the original path probability vector. Subsequently, an action mask is used to set the probability of infeasible paths such as link failures and insufficient bandwidth to 0, avoiding invalid routing decisions. Finally, the selection probability of each path is calculated using the Softmax function to ensure that the sum of probabilities is 1. The following example further illustrates the principle of S1051~S1054. When an 8-channel DFB-SA array is configured to correspond to 8 candidate routing paths, the total time window is set to 60ns (including 3 time steps, each 20ns). The pulse vectors for the 3 time steps are t1 = [4,2,5,1,3,0,4,2], t2 = [3,1,4,2,2,1,3,1], and t3 = [5,3,3,1,4,0,2,2]. The original path probability vector [4,2,4,1.33,3,0.33,3,1.67] is obtained by summing and averaging the pulse counts of the three time steps corresponding to each routing path. Then, it is detected that only the 4th and 6th paths out of the first six are infeasible (corresponding to the action mask mask=[1,1,1,0,1,0,1,1]). Therefore, the logits of these two paths are set to -∞ according to the masking rules, resulting in the path probability vector after action masking [4,2,4,-∞,3,-∞,3,1.67], thus masking invalid paths. Next, the path probability vector after action masking is input into the Softmax function to obtain the probability distribution [0.28, 0.08, 0.29, 0,0.18, 0, 0.17, 0.06]. The 3rd path with the highest probability is directly selected as the optimal routing path. Finally, an OpenFlow flow table entry is generated for the third path and sent to the relevant switches via the southbound interface. After the switches update their local flow tables, they complete the traffic forwarding according to the specified path.

[0074] The photon pulse reinforcement learning intelligent network routing method proposed in this invention has the following advantages: (1) Strong time-series feature processing capability and excellent dynamic traffic adaptability This invention employs a spiking neural network (SNN) within a spiking Actor network, leveraging the spiking and membrane potential accumulation mechanisms of LIF neurons to achieve natural expression and dynamic processing of network temporal information. Compared to traditional deep networks based on static activation functions, its built-in temporal dynamics eliminates the need for additional memory units, accurately capturing the temporal dependencies of dynamic traffic data in SDN networks (such as the temporal evolution of burst traffic and dynamic fluctuations in link load). This characteristic significantly enhances the network's sensitivity to temporal features, enabling more accurate perception of traffic change trends and link state evolution, thereby improving the adaptability and reliability of routing decisions in dynamic scenarios and avoiding link congestion problems caused by "static decisions" in traditional algorithms.

[0075] (2) Bio-inspired design with significant energy advantages Spiking neurons possess the core characteristic of being event-driven, firing only when the membrane potential exceeds a threshold. This sparsity of computational overhead reduces energy consumption in principle. This design closely aligns with the working mode of the biological brain, exhibiting distinct bio-inspired characteristics. By combining it with a photonic chip based on a DFB-SA laser, the low-power, high-parallelism physical advantages of photonic computing can be fully utilized, significantly reducing the energy consumption of routing decisions compared to traditional electronic computing architectures. Future adaptation to neuromorphic hardware runtime will further highlight its low-power, high-real-time technical features, providing a solution with both biological rationale and engineering practicality for resource-constrained scenarios such as data centers and edge nodes.

[0076] (3) Simplified weight configuration and faster response speed This invention innovatively employs an explicit MZI photonic network structure, replacing the complex triangular or rectangular network architecture of traditional photonic computing. This structure achieves optical solidification of the spiking neural network weight matrix by directly adjusting the phase of the phase shifters within the MZI units, significantly simplifying the weight configuration process and reducing its complexity. Simultaneously, the inherent parallelism of photonic computing combined with the high-efficiency transmission characteristics of explicit networks enables linear computation to be completed instantaneously in the optical domain, breaking through the speed bottleneck of traditional electronic computing. The routing decision latency of the entire system is significantly reduced (experimentally verified to be approximately nanoseconds), meeting the stringent real-time decision-making requirements of SDN networks.

[0077] (4) Strong compatibility and wide range of application scenarios. This invention strictly follows the three-layer decoupled architecture of SDN. The intelligent decision layer can be seamlessly integrated into the existing SDN network without reconstructing the data plane and control plane, and has good compatibility and scalability. On the one hand, the state space design supports the flexible integration of multiple types of network characteristics (link load, latency, hop count, etc.) and can easily adapt to different topologies (such as k=4, k=6 fat tree topologies); on the other hand, the algorithm supports the dynamic expansion of traffic scale (such as increasing traffic pairs from 64 pairs to 128 pairs), and can still maintain excellent performance under high load scenarios. With the help of optoelectronic collaborative acceleration technology, this framework can meet the routing optimization needs of large-scale data centers, computing power networks, 6G intelligent networks and other scenarios, and has broad application prospects.

[0078] (5) Excellent overall performance and good global optimization effect This invention achieves global optimization rather than local suboptimal by designing a multi-objective reward function (integrating throughput, packet loss rate, load balancing, and other metrics) and employing a pruning strategy for the PPO algorithm. Experimental results show that, compared to the traditional Dijkstra algorithm, this invention improves throughput by approximately 50%, reduces average latency by approximately 75%, and reduces packet loss rate by approximately 25%, while maintaining a load balancing score consistently above 0.88. Under high-load (200Mbps) scenarios, the performance advantages are even more significant, effectively solving the problem of uneven network resource utilization caused by the "single-objective optimization" of traditional algorithms.

[0079] The present invention also provides a photon pulse reinforcement learning intelligent network routing system for implementing the steps of the above-described photon pulse reinforcement learning intelligent network routing method. The system includes: The SDN controller is used to acquire the network status information at the current moment and construct the environment state vector at the current moment based on the network status information at the current moment. A signal modulator is used to modulate the current environmental state vector onto a multi-channel light source to obtain an optical signal carrying the current environmental state vector. An explicit MZI network is loaded with the weight matrix of a linear layer of a pre-trained pulse reinforcement learning engine, which is used to perform linear calculations on the injected optical signal to obtain the interference optical signal. The signal conditioning and conversion module is used to condition and convert the interference optical signal output by the explicit MZI network to obtain a linear electrical signal; The DFB-SA laser array contains multiple DFB-SA lasers. Each DFB-SA laser is used to simulate a neuron in the hidden layer of a pre-trained pulse actor network using the LIF neuron model. It is used to accumulate membrane potential, determine thresholds, and fire pulses on the input linear electrical signal to obtain a pulse sequence. The SDN controller is also used to generate an optimal routing path based on the pulse sequence output by the DFB-SA laser array of neurons in the last hidden layer of the simulated pre-trained pulsed Actor network using the LIF neuron model, and to generate SDN routing control instructions based on the optimal routing path to drive the data plane to perform traffic forwarding tasks.

[0080] For example, the signal conditioning and conversion module can consist of a balanced photodetector, wires, and a transimpedance amplifier, with the balanced photodetector and transimpedance amplifier electrically connected, and current summation achieved through the series connection between the wires. Furthermore, each explicit MZI network can be integrated with one signal conditioning and conversion module.

[0081] It should be noted that the principles of each part of the system have been explained in detail in the section introducing the photon pulse reinforcement learning intelligent network routing method, and will not be repeated here.

[0082] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0083] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0084] In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. While different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce a good effect.

[0085] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A routing method for a photon pulse reinforcement learning intelligent network, characterized in that, include: Obtain the network state information at the current moment, and construct the environment state vector at the current moment based on the network state information at the current moment; The current environmental state vector is modulated onto a multi-channel light source to obtain an optical signal carrying the current environmental state vector. An explicit MZI network with the weight matrix of the linear layer loaded with a pre-trained spiking reinforcement learning engine is used to perform linear calculations on the input optical signal to obtain an interference optical signal. The interference optical signal is then conditioned and converted to obtain a linear electrical signal. The pre-trained spiking reinforcement learning engine includes a pre-trained spiking Actor network, which contains an input encoding layer, N hidden layers, and an output decoding layer. Each hidden layer adopts a spiking neural network structure. The pre-trained spiking reinforcement learning engine is trained using the pruning policy gradient optimization and generalized advantage estimation method of the PPO algorithm. N is a positive integer greater than or equal to 1. Using a DFB-SA laser array to simulate neurons in the hidden layer of the pre-trained pulse actor network that employ the LIF neuron model, the input linear electrical signal is subjected to membrane potential accumulation, threshold judgment, and pulse firing to obtain a pulse sequence. Based on the pulse sequence output by the DFB-SA laser array of neurons using the LIF neuron model in the last hidden layer of the pre-trained pulsed actor network, an optimal routing path is generated to generate SDN routing control instructions based on the optimal routing path, driving the data plane to perform traffic forwarding tasks.

2. The photon pulse reinforcement learning intelligent network routing method according to claim 1, characterized in that, The network status information at the current moment includes: network topology structure information, all data requests at the current moment, the load of each link in the network topology at the current moment, and the link capacity; The step of constructing the current environment state vector based on the current network state information includes: For each data request at the current moment, the index information of the source host and the destination host is extracted from the data request, and after normalizing the index information, a host encoding vector is generated, which consists of the normalized index information of the source host and the normalized index information of the destination host. Based on the network topology structure information, each candidate path between the source host and the destination host in the data request is determined. Based on the network topology structure information, the current load of each link in the network topology, and the link capacity, the average utilization, maximum utilization, and normalized hop count of each candidate path between the source host and the destination host in the data request are determined, forming a path feature vector corresponding to each candidate path. By concatenating the host encoding vectors and path feature vectors corresponding to all the data requests at the current moment in sequence, the environment state vector at the current moment is obtained.

3. The photon pulse reinforcement learning intelligent network routing method according to claim 2, characterized in that, The data request between the source host and the destination host... Average utilization of candidate paths Maximum utilization rate and the normalized jump number The calculation formulas are as follows: ； in, E p To represent the first The set of edges of the candidate paths, the first edge between the source host and the destination host in the data request. A candidate path refers to any candidate path between the source host and the destination host in the data request. Indicates the first Links in the candidate paths The current load, For link capacity, Indicates the first The actual number of hops for each candidate path.

4. The photon pulse reinforcement learning intelligent network routing method according to claim 1, characterized in that, The pulse Actor network's method for processing the input environment state vector includes: The input encoding layer converts the input environment state vector into a pulse sequence and passes it to the first hidden layer. The input pulse sequence is linearly transformed through the linear layer in the first hidden layer to obtain the neuron input current. Then, the pulse sequence is generated based on the membrane potential accumulation and threshold triggering characteristics of the neuron using the LIF neuron model in the LIF layer of the first hidden layer and passed to the next hidden layer. The next hidden layer processes the input pulse sequence using the same principle as the first hidden layer until the last hidden layer outputs the pulse sequence to the output decoding layer. The output decoding layer generates an original output vector based on the input pulse sequence, where each element in the original output vector represents a routing path. Infeasible paths in the original output vector are filtered using an action mask to obtain an action-masked output vector. The action-masked output vector is then converted into a path selection probability distribution using a Softmax function, and the routing path with the highest probability from the path selection probability distribution is selected for output. The method for filtering infeasible paths in the original output vector using an action mask includes: for each element in the original output vector, determining whether the position index of the element is less than the number of actual available paths; if so, setting the action mask corresponding to the element to 1 to indicate that the element is valid; otherwise, setting the action mask corresponding to the element to 0 to indicate that the element is invalid. Then, elements in the original output vector whose corresponding mask is set to 0 are set to a preset minimum negative value to obtain the action-masked output vector.

5. The photon pulse reinforcement learning intelligent network routing method according to claim 1, characterized in that, The pre-trained spiking reinforcement learning engine is obtained by training an initial spiking reinforcement learning engine, wherein the initial spiking reinforcement learning engine includes an initial Critic network and an initial spiking Actor network, and the loss function of the pre-trained spiking reinforcement learning engine includes the loss function of the Critic network and the loss function of the spiking Actor network. The expression for the loss function of the Critic network is as follows: ； The expression for the loss function of the impulsive Actor network is as follows: ； ； ； ； in, Indicates batch size and status. Indicates time The environment state vector, Indicates time Environment state vector, action Indicates the state The following is the routing path output by the pulse Actor network. The set of trainable parameters is The Critic network The value estimate, The set of trainable parameters is The Critic network for state The value estimate, The set of trainable parameters is The pulse Actor network in state Select action The probability, This indicates that the pulse Actor network was in state before the parameter update. Select action The probability of; Indicates time The dominant function value, Indicates time The standardized dominance function value, The mean of the dominant function values ​​for the batch. The standard deviation of the dominant function values ​​in the batch. Indicates time The dominant function value; This represents the cutoff factor for PPO. Represents the coefficient of the entropy regularization term. Entropy represents the distribution of actions; This represents the target value calculated based on the generalized advantage estimation method; Indicates time Instant rewards This represents the long-term reward discount factor. Indicates the end of the round. Indicates the GAE trade-off factor. This indicates a cropping operation.

6. The photon pulse reinforcement learning intelligent network routing method according to claim 5, characterized in that, Indicates the execution of an action The reward received later The expression is as follows: ； ； ； ； ； ； ； ； in, This represents the preset base reward value. Indicates a traffic transfer reward. D This indicates the amount of data successfully transmitted, measured in Mbps. P drop Indicates a penalty for packet loss. Indicates the target transmission volume of the traffic. Indicates load-related penalties. U net Indicates network utilization. Indicates the average network utilization rate. Indicates the maximum utilization rate of the network. Indicates the penalty for skipping. Indicates the number of hops in the path. This represents long-term load balancing rewards. This indicates the long-term load balancing degree.

7. The photon pulse reinforcement learning intelligent network routing method according to claim 1, characterized in that, When the explicit MZI network is an explicit MZI network loaded with the weight matrix of the fully connected layer in the first hidden layer of a pre-trained spurious reinforcement learning engine, the explicit MZI network utilizing the weight matrix of the linear layer of the pre-trained spurious reinforcement learning engine performs linear calculations on the input optical signal to obtain an interference optical signal, and conditions and converts the interference optical signal to obtain a linear electrical signal, including: The light signal carrying the current environmental state vector is input into an explicit MZI network loaded with the weight matrix of the fully connected layer in the first hidden layer of the pre-trained pulse reinforcement learning engine. The explicit MZI network then performs a linear product calculation between the current environmental state vector and the weight matrix through optical interference to obtain the interference light signal. The interference optical signal is sequentially processed through photoelectric detection, current summation, and transimpedance amplification to complete photoelectric conversion and signal conditioning, resulting in a linear electrical signal.

8. The photon pulse reinforcement learning intelligent network routing method according to claim 1, characterized in that, The DFB-SA laser array comprises multiple DFB-SA lasers, each DFB-SA laser being used to simulate a neuron using a LIF neuron model in each hidden layer of the pre-trained pulsating actor network, and the multiple DFB-SA lasers being connected according to the connection relationships between neurons using the LIF neuron model in each hidden layer of the pre-trained pulsating actor network.

9. The photon pulse reinforcement learning intelligent network routing method according to claim 1, characterized in that, The step of generating an optimal routing path based on the pulse sequence output by the DFB-SA laser array of neurons using the LIF neuron model in the last hidden layer of the pre-trained pulsed actor network, and generating SDN routing control instructions based on the optimal routing path to drive the data plane to perform traffic forwarding tasks includes: The pulse sequence output by the DFB-SA laser array of neurons in the last hidden layer of the pre-trained pulse Actor network using the LIF neuron model is averaged and normalized in the time dimension to obtain the original path probability vector. The path probability vector after action masking is obtained by filtering infeasible paths from the original path probability vector based on action masking. The path probability vector after action masking is mapped to the selection probability distribution of each candidate route path by the Softmax function, and the route path with the highest probability is selected as the optimal route path. An OpenFlow flow table entry is generated based on the optimal routing path, and the OpenFlow flow table entry is sent to the data plane switch to drive the switch to forward traffic along the specified path.

10. A photon pulse reinforcement learning intelligent network routing system, characterized in that, The system for implementing the method steps of any one of claims 1-9, the system comprising: The SDN controller is used to acquire the network status information at the current moment and construct the environment state vector at the current moment based on the network status information at the current moment. A signal modulator is used to modulate the current environmental state vector onto a multi-channel light source to obtain an optical signal carrying the current environmental state vector. An explicit MZI network is loaded with the weight matrix of a linear layer of a pre-trained pulse reinforcement learning engine, which is used to perform linear calculations on the injected optical signal to obtain the interference optical signal. The signal conditioning and conversion module is used to condition and convert the interference optical signal output by the explicit MZI network to obtain a linear electrical signal; The DFB-SA laser array contains multiple DFB-SA lasers. Each DFB-SA laser is used to simulate a neuron in the hidden layer of the pre-trained pulse Actor network using the LIF neuron model. It is used to accumulate membrane potential, determine thresholds, and fire pulses on the input linear electrical signal to obtain a pulse sequence. The SDN controller is further configured to generate an optimal routing path based on the pulse sequence output by the DFB-SA laser array of neurons using the LIF neuron model in the last hidden layer of the pre-trained pulsed actor network, and to generate SDN routing control instructions based on the optimal routing path to drive the data plane to perform traffic forwarding tasks.

Images