Electric power communication optical network routing optimization method based on DQN model

Abstract

The invention discloses an electric power communication optical network routing optimization method based on a DQN model, and the method comprises the following steps: S1, collecting the information of an electric power communication optical network, and building a state space, an action space and a reward function of the electric power communication optical network routing; s2, performing offline training on the DGQN model according to the state space, the action space and the reward function to obtain a trained DGQN model; and S3, optimizing a target electric power communication optical network route according to the trained DGQN model deployment, and updating model parameters through an online learning mechanism. According to the method, the adaptive learning capability of deep reinforcement learning is utilized, so that the routing decision can dynamically adapt to the continuously changing flow load and link state in the electric power communication optical network, the defect that a traditional static routing method cannot cope with a dynamic environment is overcome, and the response speed and stability of the network are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of power communication optical network routing control, and in particular relates to a power communication optical network routing optimization method based on a DQN model. Background Art

[0002] Optical network is a communication network based on optical fiber transmission, which has the characteristics of high bandwidth and low transmission delay. Optical network is usually used in data center interconnection, Internet backbone, long-distance transmission of telecom operators and other fields. Optical network is an important part of modern communication, because it can provide high bandwidth and low latency data transmission capabilities, it is widely used in various communication services.

[0003] In optical networks, routing is the problem of how to select the transmission path of optical signals to optimize a certain performance indicator. Routing problems usually involve the following optimization goals: Minimizing latency, that is, hoping that data reaches the destination as quickly as possible, thereby reducing communication latency. Minimizing energy consumption, that is, optimizing energy efficiency and reducing the power consumption of network equipment. Load balancing, that is, ensuring that traffic on each path is evenly distributed to avoid congestion.

[0004] Optical network routing optimization for power communication is a key area that involves efficiently transmitting data and optimizing network performance to meet requirements such as high bandwidth, low latency, and low energy consumption. However, with the continuous increase in network traffic and changes in network conditions, optimizing routing decisions for optical networks has become particularly important. Traditional static routing methods often cannot fully adapt to dynamic network environments, resulting in decreased network performance and waste of resources, so an intelligent and adaptive method is needed to solve this problem.

[0005] In this context, deep reinforcement learning has gradually become a hot topic of research. Through deep reinforcement learning, we can effectively cope with complex changes in network environment. Deep reinforcement learning can not only automatically learn the optimal strategy from the environment, but also optimize decisions through a continuous trial and error process to adapt to real-time changes in network status. The core of this method is that the goal of reinforcement learning is to enable the agent to choose the best action in different states to maximize the cumulative reward by learning and optimizing strategies. This makes deep reinforcement learning show good adaptability and flexibility when dealing with large-scale and high-dimensional network routing problems.

[0006] In addition, by combining the advantages of deep learning and reinforcement learning, a more complex reward mechanism can be designed to encourage agents to consider various network performance indicators when selecting routes, such as network congestion, node load balancing, data transmission efficiency, etc. This not only improves the efficiency of routing, but also effectively reduces energy consumption, thereby achieving the goal of green communication to a certain extent. Therefore, optimizing the routing of power communication optical networks through deep reinforcement learning not only has great application value and prospects, but is also an important direction for promoting the development of intelligent communication technology. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the power communication optical network routing optimization method based on the DQN model provided by the present invention solves the problem that traditional static routing methods are often unable to fully adapt to the dynamic network environment, resulting in network performance degradation and resource waste.

[0008] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a method for optimizing routing of power communication optical network based on DQN model, comprising the following steps:

[0009] S1. Collect information about the power communication optical network and establish the state space, action space and reward function of the power communication optical network routing;

[0010] S2. Offline training of the DGQN model is performed according to the state space, action space and reward function to obtain a trained DGQN model;

[0011] S3. Optimize the target power communication optical network routing according to the trained DGQN model deployment, and update the model parameters through the online learning mechanism.

[0012] Further: in said S1, the state space of the power communication optical network includes network topology state, wavelength usage state, link quality state and traffic load state;

[0013] The action space includes routing path selection, wavelength assignment, optical power control, bandwidth allocation, priority, and congestion control;

[0014] The reward function includes delay minimization reward, energy consumption penalty, load balancing reward and bandwidth utilization reward.

[0015] Further: In S2, the DGQN model includes an input layer, a graph convolutional network layer and an output layer connected in sequence;

[0016] Among them, the input layer receives the state information of the power communication optical network, and the state information is represented in the form of a graph structure, wherein the nodes of the graph structure represent the communication nodes in the network, and the edges of the graph structure represent the connection links. The graph convolutional network layer extracts the characteristics of the state information of the power communication optical network, and the output layer outputs the Q value of each possible action in the action space through the fully connected layer, and selects the action with the highest Q value as the routing optimization action of the power communication optical network.

[0017] Further: the method for extracting the characteristics of the state information of the power communication optical network by the graph convolutional network layer is specifically:

[0018] A multi-layer graph convolution operation is used to obtain the characteristics of the state information of the power communication optical network, where the node feature matrix H generated by the l+1th layer graph convolution operation is (l+1) The specific expression is:

[0019]

[0020] In the formula, H (l) is the node feature matrix of the lth layer, is the adjacency matrix of the graph, and I is the identity matrix, A is the adjacency matrix, for The degree matrix, W (l) is the weight matrix of the lth layer, and σ is the activation function.

[0021] Further: In S2, the method for offline training of the DGQN model is specifically as follows:

[0022] Get the current state from the state space, input the current state into the DGQN model, select an action from the action space, execute the selected action according to the reward function to obtain the state of the power communication optical network after executing the action, as well as the reward, store the action, state and reward in the priority experience replay buffer for training the DGQN model, and update the parameters of the model to maximize the expected cumulative reward and reduce the correlation of the data;

[0023] During offline training of the DGQN model, the loss function of DGQN minimizes the difference between the Q value of the currently selected action and the target Q value, which is provided by the target network.

[0024] The beneficial effects of the above further scheme are as follows: the DGQN model of the present invention is combined with a priority experience replay buffer, which enables the DGQN model to quickly learn from the latest data by replaying historical state-action pairs in a shorter time, and improves the stability and adaptability of the DGQN model in a dynamic environment by periodically updating the target network.

[0025] Further: The expression of the loss function L(θ) of the DQN is specifically:

[0026]

[0027] Where θ is all the parameters of the DGQN model, θ - are all the parameters of the target network, r is the reward calculated by the reward function, γ1 is the discount factor, Q target is the target Q value, which is composed of all parameters θ of the target network - Calculated, is the expected value of state s, action a, reward r, and next state s′, s is the state of the environment at the current moment, a is the action taken by the agent in state s, s′ is the state of the agent at the next moment after executing action a in state s, a′ is an action in the set of all possible actions in the next state s′, max a′ For the next state s′, select one from all possible actions a′ that can make the target Q value Q target (s′,a′;θ - ) is the action with the largest value, Q(s,a;θ) is the Q value after the agent takes action a in state s.

[0028] The beneficial effects of the above further scheme are: the target Q value is provided by the target network, updated through a fixed delay parameter to avoid network oscillation, and by minimizing the loss function L(θ), the model continuously optimizes its Q value estimate, enabling the agent to choose the optimal action in different states.

[0029] Further: the expression of the reward r calculated by the reward function is:

[0030] r=(R1,R2,R3,R4)

[0031] In the formula, R1 is the delay minimization reward, R2 is the energy consumption penalty, R3 is the load balancing reward, and R4 is the bandwidth utilization reward. The specific expression of the delay minimization reward R1 is:

[0032] R1=-b1

[0033] Where b1 is the delay value;

[0034] The specific expression of energy consumption penalty R2 is:

[0035] R2=-α×b2

[0036] In the formula, b2 is the energy consumption value, and α is the importance weight of energy consumption, which is used to balance the relative priority of energy consumption and other optimization objectives;

[0037] The specific expression of load balancing reward R3 is:

[0038]

[0039] Where b3 is the current link load value, is the average load, β is the importance weight of the load, which is used to prevent nodes or links in the network from being overloaded;

[0040] The expression of bandwidth utilization bonus R4 is as follows:

[0041] R4=γ2×b4

[0042] Where b4 is the bandwidth utilization, and γ2 is the bandwidth utilization weight factor.

[0043] The beneficial effects of the above further schemes are: Delay minimization rewards use the negative value of delay as the main reward item to enable the agent to try to choose a path with shorter delay at each step. Energy consumption penalties set penalty values ​​for actions with high energy consumption to encourage agents to choose more energy-efficient routing solutions. Load balancing rewards can prevent certain nodes or links in the network from being overloaded. Bandwidth utilization rewards can encourage agents to use bandwidth efficiently, thereby avoiding resource waste and improving the overall utilization of the network.

[0044] Further: In S3, the method for updating the model parameters through the online learning mechanism is specifically:

[0045] (1) Adaptively update the target Q value based on the actual feedback data of the target power communication optical network and update the optical network routing optimization strategy library;

[0046] (2) In the process of selecting the action with the highest Q value for decision optimization after training the DGQN model, small batch incremental training is performed to continuously update the parameters of the model;

[0047] (3) Dynamically adjust the learning rate, exploration strategy, and discount factor of the trained DGQN model according to the power communication optical network routing optimization action;

[0048] Among them, the expression for adjusting the learning rate is specifically:

[0049]

[0050] In the formula, α t is the learning rate of the tth update, α max is the initial maximum learning rate, R t is the cumulative reward at the tth update, R max is the maximum value of the cumulative reward, R min is the minimum value of the cumulative reward, and k is the adjustment factor used to control the speed of learning rate decay;

[0051] The expression for adjusting the exploration strategy is as follows:

[0052] ∈ t =∈ min +(∈ max -∈ min )·exp(-λ·t)

[0053] In the formula, ∈ t is the exploration rate of the tth update, ∈ max is the initial maximum exploration rate, ∈ min is the initial minimum exploration rate, λ is the decay rate, which is used to control the speed at which the exploration rate decreases;

[0054] The expression for adjusting the discount factor is:

[0055]

[0056] In the formula, γ t is the discount factor for the tth update, γ max is the maximum value of the discount factor, γ min is the minimum value of the discount factor, Δ s is the current state change amplitude, Δ max is the maximum value of the state change.

[0057] Further: In S3, a fault tolerance mechanism is set in the trained DGQN model, including:

[0058] (1) Limit the Q value upper limit of high-risk actions output by the trained DGQN model;

[0059] (2) Output the Q value of each possible action through several trained DGQN models using an integrated decision or voting mechanism;

[0060] (3) Design a predefined set of low-risk fallback actions for each state as a backup plan when the model output fails or a high-risk decision is made.

[0061] The beneficial effects of the present invention are as follows: the present invention provides a power communication optical network routing optimization method based on the DQN model to effectively deal with various optimization problems in power communication optical network routing. Compared with the prior art, the present invention has the following effects:

[0062] (1) The method of the present invention utilizes the adaptive learning ability of deep reinforcement learning to enable routing decisions to dynamically adapt to the ever-changing traffic load and link status in the power communication optical network, overcoming the defect that traditional static routing methods cannot cope with dynamic environments and improving the response speed and stability of the network.

[0063] (2) The method of the present invention combines multiple optimization objectives such as delay minimization, energy consumption control, and load balancing. Through a reasonably designed reward function, it guides the intelligent agent to select the optimal path, thereby improving the overall network performance and adapting to the comprehensive requirements of power communication optical networks for high bandwidth, low latency, and low energy consumption.

[0064] (3) The method of the present invention uses a fault-tolerant mechanism and real-time monitoring to ensure that when deployed in an actual network environment, catastrophic decisions can be avoided and the high reliability of the network can be guaranteed. At the same time, the method of the present invention complies with data privacy and network security requirements, helps to comply with the laws and standards of the power communication industry, and ensures the security and compliance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A flow chart of the power communication optical network routing optimization method based on the DQN model provided by the present invention.

[0066] Figure 2 This is the structural diagram of the DGQN model.

[0067] Figure 3 Schematic diagram of offline training of the DGQN model.

[0068] Figure 4 Schematic diagram of online learning for the DGQN model. DETAILED DESCRIPTION

[0069] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0070] like Figure 1 As shown, in one embodiment of the present invention, a method for optimizing routing of an electric power communication optical network based on a DQN model comprises the following steps:

[0071] S1. Collect information about the power communication optical network and establish the state space, action space and reward function of the power communication optical network routing;

[0072] S2. Offline training of the DGQN model is performed according to the state space, action space and reward function to obtain a trained DGQN model;

[0073] S3. Optimize the target power communication optical network routing according to the trained DGQN model deployment, and update the model parameters through the online learning mechanism.

[0074] In this embodiment, the present invention is based on the classic deep reinforcement learning algorithm DQN. According to the characteristics of the power communication optical network routing, the power communication optical network routing optimization problem is modeled as a Markov decision process, and a DGQN model of the power communication optical network routing optimization method is proposed to realize the intelligent selection of data transmission paths in the optical network to meet the efficient transmission requirements under real-time dynamic network conditions; reduce network congestion and delay through adaptive routing strategies, and improve the transmission performance of the overall network; take into account the network node energy consumption and resource utilization in routing decisions, so as to achieve a more efficient and greener communication network.

[0075] In said S1, the state space of the power communication optical network includes the network topology state, wavelength usage state, link quality state and traffic load state;

[0076] Among them, the network topology is the structural information of the power communication optical network, including nodes and links. Among them, nodes include optical fiber switches and routers, and links are connections between nodes. The network topology includes not only the number and location of nodes, but also parameters such as bandwidth and delay of each link. By representing the topological structure of the network through the adjacency matrix, the intelligent agent can understand the possible paths for data transmission in the network. In the wavelength division multiplexing (WDM) system, wavelength is a key resource. The state space needs to include the current usage of wavelengths on each link, including the allocated wavelengths, the number of idle wavelengths, and the utilization rate of wavelengths. Link quality refers to the transmission performance of the link, including signal attenuation, bit error rate, delay and bandwidth. This information can be collected in real time through the network monitoring system and expressed as the value of the link. The intelligent agent needs to consider the impact of link quality in decision-making to ensure that the selected path has the best transmission performance. Traffic load is the traffic load of each node and link in the current network, including processed traffic and current traffic to be processed. Monitoring of traffic load can help the intelligent agent determine the degree of congestion in the network, so as to select a path with lower load to avoid network bottlenecks. By comprehensively considering the above factors, the design of the state space can provide a comprehensive view of the network environment, ensuring that the intelligent agent can make more informed choices when optimizing the routing of the power communication optical network, thereby improving the performance and efficiency of the overall network.

[0077] The action space includes routing path selection, wavelength assignment, optical power control, bandwidth allocation, priority, and congestion control;

[0078] Among them, routing path selection is the core part of the action space. The agent needs to select the path for data transmission according to the network topology. Wavelength resources are limited. The agent needs to allocate appropriate wavelengths on different links of each path to ensure that data can be transmitted uninterruptedly. In order to maintain the signal quality of data transmission, the agent also needs to adjust the optical power on each link. The control of optical power needs to take into account the clarity of the signal and the transmission distance to avoid the increase of data bit error rate due to excessive or weak signals. For large traffic or critical data traffic, bandwidth allocation is an important optimization operation. The agent can dynamically adjust the bandwidth allocation according to the priority of the data flow and the current load of the link. Reasonable bandwidth allocation can not only ensure the transmission quality of high-priority traffic, but also avoid the waste of bandwidth resources and improve the utilization of the overall network. When the network load is high, the agent can also optimize the transmission effect by controlling the priority of the data flow and executing congestion management measures. By selecting better paths and resources for high-priority traffic, the agent can reduce the transmission delay of critical data. Through the above multi-dimensional action space design, the agent can make optimal decisions in complex continuous and discrete combination spaces to meet the routing optimization needs of power communication optical networks. These actions as a whole form a continuous action space, which enables the agent to flexibly adjust its strategy to achieve the routing optimization goals of low latency, high efficiency and maximum resource utilization.

[0079] The reward function includes delay minimization reward, energy consumption penalty, load balancing reward and bandwidth utilization reward.

[0080] Among them, the delay minimization reward uses the negative value of the delay as the main reward item, which can make the agent try to choose a path with shorter delay in each step. The energy consumption penalty sets a penalty value for high-energy consumption actions to encourage the agent to choose a more energy-efficient routing solution. The load balancing reward can prevent certain nodes or links in the network from being overloaded. The bandwidth utilization reward can encourage the agent to use bandwidth efficiently, thereby avoiding resource waste and improving the overall utilization of the network.

[0081] like Figure 2 As shown, in S2, the DGQN model includes an input layer, a graph convolutional network layer and an output layer connected in sequence;

[0082] Among them, the input layer receives the state information of the power communication optical network, and the state information is represented in the form of a graph structure, wherein the nodes of the graph structure represent the communication nodes in the network, and the edges of the graph structure represent the connection links. The graph convolutional network layer extracts the characteristics of the state information of the power communication optical network, and the output layer outputs the Q value of each possible action in the action space through the fully connected layer, and selects the action with the highest Q value as the routing optimization action of the power communication optical network.

[0083] In this embodiment, the input layer receives the state information of the power communication optical network, including the characteristics of nodes and links, such as the network topology, wavelength usage of the link, delay and load. These feature data are represented in the form of a graph structure, where the nodes of the graph structure represent the communication nodes in the network, and the edges of the graph structure represent the connecting links and contain the characteristic attributes of each link. The input layer organizes this information into a feature vector matrix of nodes and edges, and inputs it into the graph convolutional network layer for further feature extraction.

[0084] The graph convolutional network layer is responsible for performing convolution operations on the input graph structure data to extract the characteristics of the power communication optical network. The graph convolutional network layer mainly includes multiple layers of graph convolution operations, and each layer aggregates the neighbor information of the node to capture the topological structure and node feature information in the network.

[0085] The output layer uses a fully connected layer to map the aggregated features of the nodes to the Q value of each action. Each Q value represents the expected cumulative reward for selecting the corresponding action in the current state, and the agent optimizes the decision by selecting the action with the highest Q value.

[0086] The method for extracting the features of the state information of the power communication optical network by the graph convolutional network layer is specifically as follows:

[0087] A multi-layer graph convolution operation is used to obtain the characteristics of the state information of the power communication optical network, where the node feature matrix H generated by the l+1th layer graph convolution operation is (l+1) The specific expression is:

[0088]

[0089] In the formula, H (l) is the node feature matrix of the lth layer, is the adjacency matrix of the graph, and I is the identity matrix, A is the adjacency matrix, for The degree matrix, W (l) is the weight matrix of the lth layer, and σ is the activation function.

[0090] like Figure 3 As shown, in S2, the method for offline training of the DGQN model is specifically as follows:

[0091] Get the current state from the state space, input the current state into the DGQN model, select an action from the action space, execute the selected action according to the reward function to obtain the state of the power communication optical network after executing the action, as well as the reward, store the action, state and reward in the priority experience replay buffer for training the DGQN model, and update the parameters of the model to maximize the expected cumulative reward and reduce the correlation of the data;

[0092] In this embodiment, at each time step, an action is selected according to the current state, and the action is executed to obtain the state of the power communication optical network after the action is executed, as well as the reward. The action, state, and reward are stored in the priority experience replay buffer for training the DGQN model to reduce the correlation of the data.

[0093] During offline training of the DGQN model, the loss function of DGQN minimizes the difference between the Q value of the currently selected action and the target Q value, which is provided by the target network.

[0094] The DGQN model of the present invention combines a prioritized experience replay buffer to enable the DGQN model to quickly learn from the latest data by replaying historical state-action pairs in a short time, and to improve the stability and adaptability of the DGQN model in a dynamic environment by periodically updating the target network.

[0095] The expression of the loss function L(θ) of the DQN is specifically:

[0096]

[0097] Where θ is all the parameters of the DGQN model, θ - are all the parameters of the target network, r is the reward calculated by the reward function, γ1 is the discount factor, Q target is the target Q value, which is composed of all parameters θ of the target network - Calculated, is the expected value of state s, action a, reward r, and next state s′, s is the state of the environment at the current moment, a is the action taken by the agent in state s, s′ is the state of the agent at the next moment after executing action a in state s, a′ is an action in the set of all possible actions in the next state s′, max a′ For the next state s′, select one from all possible actions a′ that can make the target Q value Q target (s′,a′;θ - ) is the action with the largest value, Q(s,a;θ) is the Q value after the agent takes action a in state s.

[0098] In this embodiment, the target Q value is provided by the target network and updated through a fixed delay parameter to avoid network oscillation. By minimizing the loss function L(θ), the model continuously optimizes its Q value estimation, enabling the agent to select the optimal action in different states.

[0099] The expression of the reward r calculated by the reward function is:

[0100] r=(R1,R2,R3,R4)

[0101] In the formula, R1 is the delay minimization reward, R2 is the energy consumption penalty, R3 is the load balancing reward, and R4 is the bandwidth utilization reward. The specific expression of the delay minimization reward R1 is:

[0102] R1=-b1

[0103] Where b1 is the delay value;

[0104] The specific expression of energy consumption penalty R2 is:

[0105] R2=-α×b2

[0106] In the formula, b2 is the energy consumption value, and α is the importance weight of energy consumption, which is used to balance the relative priority of energy consumption and other optimization objectives;

[0107] The specific expression of load balancing reward R3 is:

[0108]

[0109] Where b3 is the current link load value, is the average load, β is the importance weight of the load, which is used to prevent nodes or links in the network from being overloaded;

[0110] The expression of bandwidth utilization bonus R4 is as follows:

[0111] R4=γ2×b4

[0112] Where b4 is the bandwidth utilization, and γ2 is the bandwidth utilization weight factor.

[0113] Deploying the DGQN model to the actual power communication optical network requires addressing a series of challenges to ensure the effectiveness and security of the model. The following are several aspects that need to be considered for online deployment and learning:

[0114] (1) Data collection and updating. In online deployment, data collection is the basis for ensuring that the model adapts to real-time network conditions. The status information in the actual optical network (such as node connection, link quality, traffic load, etc.) and the corresponding rewards need to be continuously collected and updated to form high-quality training samples for DGQN to learn. To ensure the timeliness and accuracy of data collection, an automated data collection mechanism is adopted, combined with a distributed data storage system, so that the model can obtain the latest network status at any time.

[0115] (2) Adaptation to network dynamics. The optical network is a complex and dynamic system, and traffic load, link delay, and node status may change at any time. In order to adapt to this dynamic nature, the DGQN model needs to support online learning so that the routing strategy can be adjusted in real time. Specifically, the DGQN model can be combined with the "prioritized experience replay" mechanism to enable the model to quickly learn from the latest data by replaying historical state-action pairs in a short period of time. In addition, the stability and adaptability of the model in a dynamic environment can be improved by periodically updating the target network.

[0116] (3) Fault tolerance and disaster prevention. To ensure the high reliability of the optical network, the DGQN model needs to be fault-tolerant to prevent network interruption or severe performance degradation due to wrong decisions. A fault-tolerant mechanism is set up in the model to monitor the model's decisions and network status in real time. Once abnormal behavior or major errors are detected, the model is immediately rolled back to the last successful strategy to ensure the continuous stability of the network. A backup model is configured to provide an alternative solution when the main model fails, thereby improving the reliability of the network.

[0117] The main goal of online deployment is to achieve continuous learning and optimization of the model in a real-time environment, so that it can adapt to changing network conditions, thereby achieving dynamic and efficient routing. Therefore, the present invention designs a method for updating model parameters through an online learning mechanism.

[0118] like Figure 4 As shown, in S3, the method for updating the model parameters through the online learning mechanism is specifically:

[0119] (1) Adaptively update the target Q value based on the actual feedback data of the target power communication optical network and update the optical network routing optimization strategy library;

[0120] In this embodiment, the DGQN model is used to gradually optimize the strategy based on the actual feedback data of the network (such as delay, bandwidth utilization, and traffic load). For each new state-action pair, the model adaptively updates the Q value to gradually improve the routing decision and timely update the optical network routing optimization strategy library.

[0121] (2) In the process of decision optimization, the trained DGQN model selects the action with the highest Q value and performs small batch incremental training to continuously update the parameters of the model.

[0122] During the model operation, small batch incremental training is performed to continuously update the model parameters. Combined with experience replay, it can help the model balance the relationship between new and old experience, thereby avoiding overfitting to the current data.

[0123] (3) Dynamic parameter adjustment. According to the optimization actions of the power communication optical network routing, the model’s learning rate, exploration strategy (such as ∈-greedy strategy), and discount factor parameters are dynamically adjusted to improve the performance stability and optimization effect of the model in different scenarios.

[0124] Among them, the learning rate determines the size of the model update step. Dynamically adjusting the learning rate helps to balance the convergence speed and stability at different training stages. Adopting reward-based adjustment, that is, when the cumulative reward of the agent grows faster, the learning rate can be gradually reduced to avoid oscillation and stabilize convergence. When the cumulative reward grows slowly, the learning rate is appropriately increased to accelerate learning. The expression for adjusting the learning rate is:

[0125]

[0126] In the formula, α t is the learning rate of the tth update, α max is the initial maximum learning rate, R t is the cumulative reward at the tth update, R max is the maximum value of the cumulative reward, R min is the minimum value of the cumulative reward, and k is the adjustment factor used to control the speed of learning rate decay;

[0127] The ∈-greedy strategy balances exploration and utilization during training. By dynamically adjusting the value of ∈, it can better adapt to the learning needs of different stages. A linear attenuation method based on the training stage is adopted. In the early stage of training, a higher ∈ value is used (to encourage exploration); as the training progresses, the ∈ value is gradually reduced (to make more use of the learned optimal strategy). The expression for adjusting the exploration strategy is as follows:

[0128] ∈ t =∈ min +(∈ max -∈ min )·exp(-λ·t)

[0129] In the formula, ∈ t is the exploration rate of the tth update, ∈ max is the initial maximum exploration rate, ∈ min is the initial minimum exploration rate, λ is the decay rate, which is used to control the speed at which the exploration rate decreases;

[0130] The discount factor γ determines the importance that the agent places on future rewards. In a dynamic environment, adjusting the discount factor according to the characteristics of the task can better balance short-term and long-term benefits. Using a state-based dynamic adjustment method, if the network state changes rapidly (for example, links fluctuate frequently), use a smaller discount factor to emphasize short-term rewards; if the network state is relatively stable, increase the discount factor appropriately to focus on long-term optimization goals. The specific expression for adjusting the discount factor is:

[0131]

[0132] In the formula, γ t is the discount factor for the tth update, γ max is the maximum value of the discount factor, γ min is the minimum value of the discount factor, Δ s is the current state change amplitude, Δ max is the maximum value of the state change.

[0133] In S3, a fault tolerance mechanism is set in the trained DGQN model, including:

[0134] (1) Limit the Q value upper limit of high-risk actions output by the trained DGQN model to prevent extreme decisions from being made in special circumstances;

[0135] (2) Using several trained DGQN models to output the Q value of each possible action using an integrated decision or voting mechanism to improve the robustness of model decision making and avoid the risk of single model failure;

[0136] (3) A predefined set of low-risk fallback actions is designed for each state as a backup plan when the model output fails or a high-risk decision is made, providing a fallback plan for the model's incorrect decisions and ensuring network security.

[0137] The beneficial effects of the present invention are as follows: the present invention provides a power communication optical network routing optimization method based on the DQN model to effectively deal with various optimization problems in power communication optical network routing. Compared with the prior art, the present invention has the following effects:

[0138] (1) The method of the present invention utilizes the adaptive learning ability of deep reinforcement learning to enable routing decisions to dynamically adapt to the ever-changing traffic load and link status in the power communication optical network, overcoming the defect that traditional static routing methods cannot cope with dynamic environments and improving the response speed and stability of the network.

[0139] (2) The method of the present invention combines multiple optimization objectives such as delay minimization, energy consumption control, and load balancing. Through a reasonably designed reward function, it guides the intelligent agent to select the optimal path, thereby improving the overall network performance and adapting to the comprehensive requirements of power communication optical networks for high bandwidth, low latency, and low energy consumption.

[0140] (3) The method of the present invention uses a fault-tolerant mechanism and real-time monitoring to ensure that when deployed in an actual network environment, catastrophic decisions can be avoided and the high reliability of the network can be guaranteed. At the same time, the method of the present invention complies with data privacy and network security requirements, helps to comply with the laws and standards of the power communication industry, and ensures the security and compliance of the system.

[0141] In the description of the present invention, it is necessary to understand that the orientation or positional relationship indicated by the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", and "third" are used only for descriptive purposes, and cannot be understood as indicating or implying the relative importance or the number of implicitly specified technical features. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of the features.

Claims

1. A method for optimizing routing in an electric power communication optical network based on a DQN model, characterized in that: The following steps are involved: S1. Collect information about the power communication optical network and establish the state space, action space and reward function of the power communication optical network routing; S2. Offline training of the DGQN model is performed according to the state space, action space and reward function to obtain a trained DGQN model; S3. Optimize the target power communication optical network routing according to the trained DGQN model deployment, and update the model parameters through the online learning mechanism.

2. The method for optimizing routing of power communication optical network based on DQN model according to claim 1, characterized in that: In said S1, the state space of the power communication optical network includes the network topology state, wavelength usage state, link quality state and traffic load state; The action space includes routing path selection, wavelength assignment, optical power control, bandwidth allocation, priority, and congestion control; The reward function includes delay minimization reward, energy consumption penalty, load balancing reward and bandwidth utilization reward.

3. The method for optimizing routing of power communication optical network based on DQN model according to claim 1, characterized in that: In S2, the DGQN model includes an input layer, a graph convolutional network layer, and an output layer connected in sequence; Among them, the input layer receives the state information of the power communication optical network, and the state information is represented in the form of a graph structure, wherein the nodes of the graph structure represent the communication nodes in the network, and the edges of the graph structure represent the connection links. The graph convolutional network layer extracts the characteristics of the state information of the power communication optical network, and the output layer outputs the Q value of each possible action in the action space through the fully connected layer, and selects the action with the highest Q value as the routing optimization action of the power communication optical network.

4. The method for optimizing routing of power communication optical network based on DQN model according to claim 3 is characterized in that: The method for extracting the features of the state information of the power communication optical network by the graph convolutional network layer is specifically as follows: A multi-layer graph convolution operation is used to obtain the characteristics of the state information of the power communication optical network, where the node feature matrix H generated by the l+1th layer graph convolution operation is (l+1) The specific expression is: In the formula, H (l) is the node feature matrix of the lth layer, is the adjacency matrix of the graph, and I is the identity matrix, A is the adjacency matrix, for The degree matrix, W (l) is the weight matrix of the lth layer, and σ is the activation function.

5. The method for optimizing routing of power communication optical network based on DQN model according to claim 3 is characterized in that: In S2, the method for offline training of the DGQN model is specifically as follows: Get the current state from the state space, input the current state into the DGQN model, select an action from the action space, execute the selected action according to the reward function to obtain the state of the power communication optical network after executing the action, as well as the reward, store the action, state and reward in the priority experience replay buffer for training the DGQN model, and update the parameters of the model to maximize the expected cumulative reward and reduce the correlation of the data; During offline training of the DGQN model, the loss function of DGQN minimizes the difference between the Q value of the currently selected action and the target Q value, which is provided by the target network.

6. The method for optimizing routing of power communication optical network based on DQN model according to claim 5 is characterized in that: The expression of the loss function L(θ) of the DQN is specifically: In the formula, θ is all the parameters of the DGQN model, θ - are all the parameters of the target network, r is the reward calculated by the reward function, γ1 is the discount factor, Q target is the target Q value, which is composed of all parameters θ of the target network - Calculated, is the expected value of state s, action a, reward r, and next state s′, s is the state of the environment at the current moment, a is the action taken by the agent in state s, s′ is the state of the agent at the next moment after executing action a in state s, a′ is an action in the set of all possible actions in the next state s′, max a′ For the next state s′, select one from all possible actions a′ that can make the target Q value Q target (s′,a′;θ - ) is the action with the largest value, Q(s,a;θ) is the Q value after the agent takes action a in state s.

7. The method for optimizing routing of power communication optical network based on DQN model according to claim 6 is characterized in that: The expression of the reward r calculated by the reward function is: r=(R1,R2,R3,R4) In the formula, R1 is the delay minimization reward, R2 is the energy consumption penalty, R3 is the load balancing reward, and R4 is the bandwidth utilization reward. The specific expression of the delay minimization reward R1 is: R1=-b1 Where b1 is the delay value; The specific expression of energy consumption penalty R2 is: R2=-α×b2 In the formula, b2 is the energy consumption value, and α is the importance weight of energy consumption, which is used to balance the relative priority of energy consumption and other optimization objectives; The specific expression of load balancing reward R3 is: Where b3 is the current link load value, is the average load, β is the importance weight of the load, which is used to prevent nodes or links in the network from being overloaded; The expression of bandwidth utilization bonus R4 is as follows: R4=γ2×b4 Where b4 is the bandwidth utilization, and γ2 is the bandwidth utilization weight factor.

8. The method for optimizing routing of power communication optical network based on DQN model according to claim 1, characterized in that: In S3, the method for updating the model parameters through the online learning mechanism is specifically as follows: (1) Adaptively update the target Q value based on the actual feedback data of the target power communication optical network and update the optical network routing optimization strategy library; (2) In the process of selecting the action with the highest Q value for decision optimization after training the DGQN model, small batch incremental training is performed to continuously update the parameters of the model; (3) Dynamically adjust the learning rate, exploration strategy, and discount factor of the trained DGQN model according to the power communication optical network routing optimization action; Among them, the expression for adjusting the learning rate is specifically: In the formula, α t is the learning rate of the tth update, α max is the initial maximum learning rate, R t is the cumulative reward at the tth update, R max is the maximum value of the cumulative reward, R min is the minimum value of the cumulative reward, and k is the adjustment factor used to control the speed of learning rate decay; The expression for adjusting the exploration strategy is as follows: ∈ t =∈ min +(∈ max -∈ min )·exp(-λ·t) In the formula, ∈ t is the exploration rate of the tth update, ∈ max is the initial maximum exploration rate, ∈ min is the initial minimum exploration rate, λ is the decay rate, which is used to control the speed at which the exploration rate decreases; The expression for adjusting the discount factor is: In the formula, γ t is the discount factor for the tth update, γ max is the maximum value of the discount factor, γ min is the minimum value of the discount factor, Δ s is the current state change amplitude, Δ max is the maximum value of the state change.

9. The method for optimizing routing of power communication optical network based on DQN model according to claim 1, characterized in that: In S3, a fault tolerance mechanism is set in the trained DGQN model, including: (1) Limit the Q value upper limit of high-risk actions output by the trained DGQN model; (2) Output the Q value of each possible action through several trained DGQN models using an integrated decision or voting mechanism; (3) Design a predefined set of low-risk fallback actions for each state as a backup plan when the model output fails or a high-risk decision is made.

Images