Self-adaptive fault-tolerant routing planning method for faulty on-chip optical network based on Q-learning algorithm

Abstract

The invention provides a Q-learning algorithm-based adaptive fault-tolerant routing planning method for a faulty on-chip optical network. The method comprises the following steps: S1, constructing a network model according to the state of an optical router in the network and a network topology structure, and defining a network path reward value evaluation model; s2, calculating a corresponding path reward value from a current node to an adjacent node according to the network path reward value evaluation model, determining an optimal next node from the path reward value, updating a Q table, and enabling the node to be the current node; s3, if the current node is a target node, ending the calculation of this round, and obtaining a planned path and a reward value of the path; if the current node is located in the fault domain, ending the calculation; if not, repeating the steps S2-S3; and S4, obtaining a plurality of planned paths according to a preset learning round, and determining an optimal path from the planned paths. According to the method provided by the invention, the optimal path is selected from the paths which are prevented from passing through the fault domain through the Q-learning algorithm, so that the network performance can be effectively improved.

Description

Technical Field

[0001] The invention belongs to the technical field of dynamic routing planning, and in particular relates to an adaptive fault-tolerant routing planning method for a faulty on-chip optical network based on a Q-learning algorithm. Background Art

[0002] With the breakthroughs in silicon-based integrated device technology and silicon-photonics technology, on-chip optical network systems have attracted widespread attention due to their high bandwidth, low latency, and low power consumption compared to traditional on-chip systems. As the main component of on-chip optical networks, the performance of optical router structures directly affects the performance of on-chip networks. Microring resonators are the core components of optical routers. Due to temperature effects or manufacturing defects, microring resonators are very likely to cause device failures due to resonant wavelength drift, thereby causing failures in the entire optical router. When an optical router fails, it will further lead to data mistransmission and loss during system communication, reducing the security and reliability of the optical on-chip network system. Summary of the invention

[0003] In order to solve the above problems existing in the prior art, the purpose of the present invention is to provide a method for adaptive fault-tolerant routing planning for faulty on-chip optical networks based on a Q-learning algorithm:

[0004] The present invention provides a Q-learning algorithm-based adaptive fault-tolerant routing planning method for faulty on-chip optical networks, comprising:

[0005] S1: According to the status of the optical routers in the network and the network topology, a network model is constructed, and a network path reward value evaluation model is defined to initialize the Q value;

[0006] The network model includes nodes of multiple optical routers and bidirectional links between nodes, paths from source nodes to target nodes including multiple data links and fault domains formed by faulty nodes;

[0007] The network path reward value evaluation model includes the insertion loss between nodes representing the bidirectional link and the different insertion losses caused by the different states of the optical routers, and the corresponding reward values ​​are set according to the different insertion losses;

[0008] S2: Calculate the corresponding path reward value from the current node to its adjacent node according to the network path reward value evaluation model, determine the optimal next node, update the Q table, and set the node as the current node;

[0009] S3: Determine whether the current node is the target node or is located in the fault domain;

[0010] If none, repeat steps S2-S3

[0011] If the current node is the target node, end this round of calculation, and obtain a planned path and its reward value according to the Q table;

[0012] If the current node is in the fault domain, end this round of calculation;

[0013] The reward value of the planned path is the sum of the reward values ​​corresponding to the selected nodes during the path planning process.

[0014] S4: obtaining a plurality of planned paths according to a preset learning round, and selecting a planned path with a maximum reward value as the optimal path according to the size of the reward value of the planned path;

[0015] In one embodiment of the present invention, path planning is performed according to the Q-learning algorithm to obtain the source node v s To the target node v d The best planning path, where the source node v s With the target node v d Not a faulty node, including:

[0016] Step 1: Get the current network topology information, including:

[0017] The nodes of the multiple optical routers forming the unit and the bidirectional links between the nodes, the paths from the source node to the target node including multiple paths consisting of data links and the status of the optical routers;

[0018] Step 2: Based on the network topology structure in step 1 and the state of the optical routers in the network, a network model is constructed, the maximum number of routing hops h is set according to the network size, and the corresponding reward value is set according to the insertion loss caused by the different states of the optical routers. The smaller the insertion loss, the greater the reward value.

[0019] Step 3: Set the source node v s As the current node v k , initialize Q-learning parameters and Q values ​​of all nodes;

[0020] Step 4: Calculate the corresponding path reward value from the current node to its adjacent nodes based on the network path reward value evaluation, determine the optimal next node, update the Q table, update the Q value, and use the node as the new current node v k ;

[0021] Step 5: Determine node v k+1 Whether it is the target node;

[0022] If the node v k+1If it is the target node d, then this round of learning ends, and according to the Q table, a planned path and its reward value r are obtained. q ;

[0023] Step 6: Determine node v k+1 Is it located in the fault domain or the current routing hop count h>=h max ;

[0024] If yes, then end this round of learning; if no, repeat steps 4 to 6;

[0025] Step 7: Obtain several planned paths according to the preset learning rounds, and determine the sum of the reward values ​​r of the several planned paths q The size of the reward value r of the planned path is selected q The largest planned path is taken as the optimal path;

[0026] In one embodiment of the present invention, the Q-learning learning parameters include: learning round q max , learning rate α, discount factor γ.

[0027] In one embodiment of the present invention, the reward value of the planned path is the sum of the reward values ​​r corresponding to the selected nodes in the path planning process;

[0028] In one embodiment of the present invention, the state of the optical router can be divided into a normal state and a fault state;

[0029] When the state of the optical router is normal, that is, the input optical signal can be output according to the designated port by controlling the optical switch in the optical router, the node whose state of the optical router is normal is a normal node;

[0030] When the state of the optical router is a fault state, it will cause serious insertion loss of input light or fail to transmit along the specified path. The node whose state of the optical router is a fault state is a faulty node.

[0031] In one embodiment of the present invention, if there is only one normal node left among the nodes adjacent to a normal node, then the node is a dangerous node;

[0032] In one embodiment of the present invention, if all nodes adjacent to a normal node are faulty nodes, the node is in an unavailable state, and the node can be regarded as a faulty node;

[0033] In one embodiment of the present invention, adjacent faulty nodes and dangerous nodes are linked into a fault domain, where the fault domain can be divided into convex or rectangular faults;

[0034] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of a system of a Q-learning algorithm provided by an embodiment of the present invention;

[0036] Figure 2 It is a schematic diagram of a 8×8 mesh on-chip optical network simulation provided by an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of an on-chip optical network IP node simulation provided by an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of a 5-port optical routing node simulation provided by an embodiment of the present invention;

[0039] Figure 5 This is a fault domain schematic diagram provided by an embodiment of the present invention;

[0040] Figure 6 is a path selection schematic diagram provided by an embodiment of the present invention;

[0041] Figure 7 is a specific flow chart of a Q-learning algorithm provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0042] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of a Q-learning algorithm-based adaptive fault-tolerant routing planning method for faulty on-chip optical networks proposed in accordance with the present invention, in combination with the accompanying drawings and specific implementation methods.

[0043] The above and other technical contents, features and effects of the present invention are clearly presented in the following detailed description of the specific implementation modes in conjunction with the accompanying drawings. Through the description of the specific implementation modes, the technical means and effects adopted by the present invention to achieve the predetermined purpose can be more deeply and specifically understood. However, the attached drawings are only for reference and explanation purposes and are not used to limit the technical solutions of the present invention.

[0044] The traditional on-chip optical network mainly consists of an optical network layer and an electrical control network layer. The optical network layer is formed by connecting optical routers and waveguides. In order to achieve control purposes, the electronic control network is overlapped with the optical network. Packet transmission is carried out in a circuit switching manner. Before each payload transmission, a setup control packet is issued and sent. It will be routed at the electrical layer according to the routing algorithm. After the setup control packet reaches the target node from the source node, the complete optical path configuration is successful. Then, the payload will be converted into an optical signal and input into the optical network layer from the source node IP node, transmitted along the configured optical path, and finally output from the target node IP node. After all optical information is transmitted, the entire configuration is completed.

[0045] Embodiment 1

[0046] Establishing an N×N mesh optical network simulation model in network simulation software Based on the N×N mesh network and a 5-port router, the Q_learning algorithm-based on-chip optical network adaptive fault-tolerant routing planning method of this embodiment is specifically described.

[0047] Figure 1 The flowchart of a method for adaptive fault-tolerant routing planning for a faulty on-chip optical network based on a Q-learning algorithm provided by an embodiment of the present invention. As shown in the figure, the method of the present invention includes:

[0048] S1: According to the status of the optical routers in the network and the network topology, a network model is constructed, and a network path reward value evaluation model is defined to initialize the Q value.

[0049] The network model includes nodes that form a plurality of optical router units and bidirectional links between nodes, and a path from a source node to a target node includes a plurality of paths composed of data links.

[0050] The network path reward value evaluation model includes the insertion loss between nodes representing the bidirectional link and the different insertion losses caused by the different states of the optical routers. The corresponding reward values ​​are set according to the different insertion losses.

[0051] S2: Calculate the corresponding path reward value from the current node to its adjacent node according to the network path reward value evaluation model, determine the optimal next node therefrom, update the Q table, and make the node the current node.

[0052] S3: Determine whether the current node is the target node or is located in the fault domain.

[0053] If not, repeat steps S2-S3;

[0054] If the current node is the target node, end this round of calculation, and obtain a planned path and its reward value according to the Q table;

[0055] If the current node is in the fault domain, end this round of calculation.

[0056] refer to Figure 2 , which is a schematic diagram of a 8×8mesh on-chip optical network simulation provided by an embodiment of the present invention; Figure 3 , which is a schematic diagram of an on-chip optical network IP node simulation schematic diagram provided by an embodiment of the present invention; reference Figure 4 , which is a schematic diagram of a 5-port optical routing node simulation provided by an embodiment of the present invention; Figure 5 , which is a schematic diagram of a fault domain provided by an embodiment of the present invention; refer to Figure 6 , which is a path selection schematic diagram provided by an embodiment of the present invention.

[0057] One port of the 5-port optical routing node is used to connect to the IP node, and the remaining four ports are used to connect to other optical routing nodes.

[0058] refer to Figure 7 1 is a specific flow chart of a Q-learning algorithm provided by an embodiment of the present invention. As shown in the figure, path planning is performed according to the Q-learning algorithm to obtain the source node v s To the target node v d The best planning path, where the source node v s With the target node v d Not a faulty node or a failed node, including:

[0059] Step 1: Set the source node v s As the current node v k， Initialize the Q value and learning parameters in Q-learning, where the Q value is Q k (s,a), represents the node v k In state s, the node in direction a is selected as the path reward value of the next node.

[0060] Among them, state s represents node v k Respectively with the target node v d and each faulty node v fi The matrix composed of the coordinate differences of is as follows.

[0061] s=(x d -x k ,y d -y k :x f1 -x k ,y f1 -y k :……:x fi -x k ,yfi -y k :……:x fn -x k ,y fn -y k )

[0062] Among them, x k ,y k are the horizontal and vertical coordinates of node vk respectively; x d ,y d The target nodes v d Horizontal and vertical coordinates; x fi ,y fi For each fault node v f The horizontal and vertical axes, n, are the number of faulty nodes.

[0063] Among them, the Q value a represents the node v k In the direction leading to the next hop node, the optical router used in this embodiment is a five-port optical router, one port of which is connected to the IP node of the circuit layer, and the other four ports are connected to other routers of the optical path layer, so in this embodiment, the value range of a is 1-4.

[0064] In this embodiment, the step of initializing the Q value is to set the Q value of each node corresponding to all directions to a random value between 0 and 5.

[0065] The Q-learning parameters include: the maximum number of learning rounds q max , probability value ε, learning rate α, discount factor γ. In this embodiment, probability value ε=0.1, learning rate α=0.1, discount factor γ=0.95.

[0066] Step 2: According to the Q value, select the next node v based on the random probability strategy k+1 , and get the reward value r for selecting this node t+1 .

[0067] Specifically, at node v k The Q values ​​corresponding to different directions are generally different. By comparison, different degrees of selection intention can be obtained. The random probability strategy is to select the next node. When the random probability rand < ε, according to the current Q value, select the best next node v from the nodes adjacent to the current node. k+1 ; When the random probability rand>=ε, at the current node v k The adjacent nodes randomly select one of the nodes as the next node v k+1 .

[0068] Step 3: Based on the selected node v k+1, update the Q value, update the total reward value r

[0069] Specifically, in this embodiment, the Q value update formula is as follows:

[0070] Q k (s,a)=Q k (s,a)+α(r k +γmax(Q k (s,a))-Q k (s,a))

[0071] Among them, α represents the learning rate, which reflects the convergence speed of the Q-learning algorithm; γ represents the discount coefficient, which reflects the influence of future feedback on the current choice. max(Q k (s,a) indicates that the target router node is v d When the current node v k Select the node in direction a as the next hop node with the maximum path reward value.

[0072] Update the current road reward value sum r = r + r k+1 , where r k+1 For node v k+1 The corresponding reward value.

[0073] Step 4: Determine node v k+1 Whether it is the target node;

[0074] If the node v k+1 If it is the target node d, then this round of learning ends, and a planned path and a reward value of the planned path are obtained according to the Q table, where r q =r.

[0075] Step 5: Determine node v k+1 Is it located in the fault domain or the current routing hop count h>=h max ;

[0076] If yes, then end this round of learning; if no, repeat steps 2 to 5.

[0077] Step 6: Obtain several planned paths according to the preset learning rounds, and determine the sum of the reward values ​​r of the several planned paths q The size of the reward value r of the planned path is selected q The largest planned path is taken as the optimal path.

[0078] In one embodiment of the present invention, the Q-learning learning parameters include: learning round q_n, learning rate α, and discount factor γ.

[0079] In one embodiment of the present invention, the reward value of the planned path is the sum of the reward values ​​r corresponding to the selected nodes in the path planning process.

[0080] In one embodiment of the present invention, the state of the optical router can be divided into a normal state and a fault state;

[0081] When the state of the optical router is normal, the input optical signal can be output according to the designated port by controlling the optical switch in the optical router, and the node whose state of the optical router is normal is a normal node;

[0082] When the state of the optical router is a fault state, it will cause serious insertion loss of input light or fail to transmit along the specified path. The node whose state of the optical router is a fault state is a faulty node.

[0083] In one embodiment of the present invention, if there is only one normal node left among the nodes adjacent to a normal node, then the node is a dangerous node;

[0084] In one embodiment of the present invention, if all nodes adjacent to a normal node are faulty nodes, the node is in an unavailable state, and the node can be regarded as a faulty node;

[0085] In one embodiment of the present invention, adjacent faulty nodes and dangerous nodes are connected to form a fault domain, wherein the fault domain can be divided into a convex or rectangular fault.

Claims

1. The present invention provides a Q-learning algorithm-based adaptive fault-tolerant routing planning method for faulty on-chip optical networks, comprising: S1: According to the status of the optical routers in the network and the network topology, a network model is constructed, and a network path reward value evaluation model is defined, and the Q value and its Q-learning parameters are initialized; S2: Calculate the corresponding path reward value from the current node to its adjacent node according to the network path reward value evaluation model, determine the optimal next node, update the Q value, and set the node as the current node; S3: Determine whether the current node is the target node or is located in the fault domain; If not, repeat steps S2-S3; If the current node is the target node, end this round of calculation, and obtain a planned path and its reward value according to the Q table; If the current node is in the fault domain, end this round of calculation; S4: Obtain a plurality of planned paths according to a preset learning round, and select the optimal path from them.

2. The method for adaptive routing planning of optical chip networks based on Q-learning algorithm according to claim 1, characterized in that: The network model includes nodes including multiple optical router components and bidirectional links between nodes in the network model. The path from the source node to the target node includes multiple paths composed of data links and a fault domain formed by a faulty node.

3. The method for adaptive routing planning of optical chip networks based on Q-learning algorithm according to claim 1, characterized in that: The network path reward value evaluation model includes the insertion loss representing the bidirectional link between nodes and different insertion losses caused by different states of optical routers, and corresponding reward values ​​are set according to different insertion losses.

4. The method for adaptive fault-tolerant routing planning for faulty on-chip optical networks based on Q-learning algorithm according to claim 1, characterized in that: The status of an optical router can be divided into normal status and fault status: Among them, when the state of the optical router is normal, the input optical signal can be output according to the specified port by controlling the optical switch in the optical router, and the node whose state of the optical router is normal is a normal node; when the state of the optical router is faulty, it will cause serious insertion loss of the input light or fail to transmit according to the specified path, and the node whose state of the optical router is faulty is a faulty node.

5. The method for adaptive fault-tolerant routing planning for faulty on-chip optical networks based on Q-learning algorithm according to claim 1, characterized in that: The reward value of the planned path is the sum of the reward values ​​r corresponding to the selected nodes during the path planning process.

6. The method for adaptive routing planning of optical chip networks based on Q-learning algorithm according to claim 1, characterized in that: The fault domain is an area where adjacent faulty nodes and dangerous nodes are connected. If there is only one normal node left among the nodes adjacent to a normal node, then the node is a dangerous node; if all the nodes adjacent to a normal node are faulty nodes, then the node is in an unavailable state and can be regarded as a faulty node; adjacent faulty nodes and dangerous nodes will be connected into a fault domain, where the fault domain can be divided into convex or rectangular faults.

7. The method for adaptive fault-tolerant routing planning for faulty on-chip optical networks based on Q-learning algorithm according to claim 1, characterized in that: A plurality of planned paths are obtained according to a preset learning round, and a planned path with the largest reward value of the planned path is selected as the optimal path.

8. The method for adaptive routing planning of optical chip networks based on Q-learning algorithm according to claim 1, characterized in that: The Q value is Q k (s,a), represents the node v k In state s, select the node in direction a as the path reward value of the next node; state s represents node v k The matrix formed by the coordinate difference between the target node and each fault node; a represents the node v k The direction to the next hop node.

9. The method for adaptive routing planning of optical chip networks based on Q-learning algorithm according to claim 1, characterized in that: The Q value is updated by the following formula. Q k (s,a)=Q k (s,a)+α(r k +γmax(Q k (s,a))-Q k (s,a)) Among them, α represents the learning rate, which reflects the convergence speed of the Q-learning algorithm; γ represents the discount coefficient, which reflects the influence of future feedback on the current choice. max(Q k (s,a) indicates that the target router node is v d When the current node v k Select the node in direction a as the next hop node with the maximum path reward value.

10. The method for adaptive fault-tolerant routing planning for faulty on-chip optical networks based on Q-learning algorithm according to claim 8, characterized in that: Node v k Q value Q k The state s formula in (s,a) can be expressed by the following formula. s=(x d -x k ,and d -and k :x f1 -x k ,and f1 -and k :……:x fi -x k ,and fi -and k :……:x fn -x k ,and fn -and k ) Among them, x k ,y k The nodes v k Horizontal and vertical coordinates; x d ,y d The target nodes v d Horizontal and vertical coordinates; x fi ,y fi For each fault node v f On the horizontal and vertical axes, n is the number of faulty nodes.

Images