X-Structure Obstacle Avoidance Steiner Minimum Tree Method Based on Deep Reinforcement Learning

Abstract

The present invention relates to an X-structure obstacle-avoiding Steiner minimum tree method based on deep reinforcement learning, which includes the following three stages: a preprocessing stage, simplifying the multi-pin netlist problem based on the minimum spanning tree algorithm to construct an MST; a main routing stage, using the MST to establish the environment for the training of the agent in DRL, and finally using the trained agent to complete the routing to obtain an XSMT, and then completing the obstacle avoidance according to the obstacle-avoiding strategy to obtain an OAXSMT; a post-processing stage, using the DFS algorithm to break the loop of the OAXSMT to obtain the routing result and calculating the wire length of the routing result. The present invention can quickly and effectively obtain the optimal routing result and improve the physical design efficiency of very large scale integrated circuits.

Description

Technical Field

[0001] The present invention relates to an X-structure obstacle-avoiding Steiner minimum tree method based on deep reinforcement learning. Background Art

[0002] With the increasing complexity of the physical design of very large scale integrated circuits (VLSIs), the number of transistors per unit area has increased exponentially with technological progress. The physical design of VLSIs requires more effective electronic design automation tools. As an NP-hard problem, global routing is an important step in the physical design of VLSIs. The problems involved are highly difficult. With the continuous integration of many reusable components such as macro modules, the demand for the obstacle-avoiding ability of XSMT in global routing is increasing. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide an X-structure obstacle-avoiding Steiner minimum tree method based on deep reinforcement learning, which can quickly and effectively obtain an optimal routing result and improve the efficiency of the physical design of very large scale integrated circuits.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions:

[0005] The present invention has the following beneficial effects compared with the prior art:

[0006] An X-structure obstacle-avoiding Steiner minimum tree method based on deep reinforcement learning includes the following three stages:

[0007] The preprocessing stage simplifies the multi-pin netlist problem based on the minimum spanning tree algorithm to construct an MST.

[0008] The main routing stage uses the MST to establish the environment for the training of the agent in DRL. Finally, the trained agent is used to complete the routing to obtain XSMT, and then the obstacle-avoiding strategy is used to complete the obstacle avoidance to obtain OAXSMT.

[0009] The postprocessing stage uses the DFS algorithm to break the loops of OAXSMT to obtain the routing result and calculates the wire length of the routing result.

[0010] Further, in the preprocessing stage, the basic information of the netlist extracted includes the set of pin coordinates, the set of obstacle coordinates, the lattice mapping table constructed using the pin coordinates, and the Prim algorithm is used to construct the MST.

[0011] Further, the agent adopts a deep Q-network. By simulating the interaction between the agent and the environment, different rewards are obtained from the environment to train the agent to find the solution with the highest reward. Specifically:

[0012] For each two-pin problem, the environment provides state information to the network.

[0013] Then, the agent estimates the Q-values for each possible action (a0, a1, a2, a3) of the state;

[0014] Finally, according to the ε-greedy algorithm, the action to be executed next is selected and the agent continues to observe the next state in the environment.

[0015] Furthermore, the preset elements in the deep Q-network include

[0016] State space, the state is defined as a 4D vector. The first two elements are the rectangular coordinates of the starting point of the agent's current state, and the last two elements are the coordinates of the end point;

[0017] Action space, the action represents the wiring method between adjacent nodes in the MST;

[0018] Reward, the reward is determined according to the current wiring line sharing and wire length reduction obtained by the selected action, as shown in the following formula (3):

[0019]

[0020] Furthermore, the ε-greedy algorithm is specifically as follows: randomly select a connection action from the four actions to connect the pins, so that the agent is more daring to try possible better solutions that have not been discovered. The greedy strategy is to select the wiring action with the largest Q-value according to the existing experience, and this strategy is defined as the following formula (4):

[0021]

[0022] Furthermore, the deep Q-network contains 2 fully connected layers, where the hidden layer contains 32 neurons, followed by a rectified linear unit activation layer; the size of the input layer is 4, which is the same as the length of the state vector; the size of the output layer is 4, which is the same as the size of the action space.

[0023] Furthermore, the obstacle avoidance strategy is specifically as follows:

[0024] According to the XMST, first perform an obstacle penetration judgment on the XMST according to the obstacle penetration judgment strategy, and then perform a simulated obstacle avoidance to judge the obstacles that the obstacle avoidance line may pass through, so as to obtain all the obstacles that need to be considered when performing obstacle avoidance on the XMST. Then, the obstacle avoidance order is determined through iteration until a suitable obstacle avoidance order is found;

[0025] Then, virtual Steiner points are selected according to the preset requirements and added to the XMST;

[0026] Finally, the obstacle avoidance is completed according to the given four basic connection actions.

[0027] Further, for the penetration obstacle judgment strategy, the basic connection actions between points in XSMT are judged for obstacle penetration. First, the connection actions are extracted into individual line segments, and then the relationship between the line segments and obstacles is judged by the following method:

[0028] Judge whether the endpoints of the line segment are inside the rectangular obstacle. If one of the endpoints is inside, it penetrates this obstacle;

[0029] If neither endpoint is inside the obstacle, judge whether the rectangle O1 formed by the line segment intersects with the rectangular obstacle O2. If they do not intersect, then AS does not penetrate this obstacle O2;

[0030] If the above two rectangles intersect, judge whether the line segment AS intersects with the main diagonal and the secondary diagonal of the rectangular obstacle, that is, the blue line part. If they intersect, it penetrates this obstacle, otherwise it does not penetrate this obstacle.

[0031] Further, the specific simulation of obstacle bypassing is as follows: For the obstacles that have been judged to be penetrated currently, according to all the obstacle bypassing orders, a virtual Steiner point is selected for obstacle bypassing once. If this simulation of obstacle bypassing is to find an obstacle bypassing route and causes penetration of a new obstacle, then the newly penetrated obstacle is added to the obstacle bypassing set. If this simulation of obstacle bypassing is to find a suitable obstacle bypassing order, then the current obstacle bypassing order is returned when the simulation of obstacle bypassing is successful.

[0032] Further, the specific loop breaking process is as follows: Add the intersection points of all line segments, virtual Steiner points, and pin points to the point set, store the connection relationships between points using an adjacency matrix, traverse the entire graph through DFS, and use the visit array to record the access records. When a point that has been visited is repeatedly visited, that is, a loop appears, delete this edge, thereby completing the loop breaking of the circuit.

[0033] The present invention can quickly and effectively obtain the optimal wiring result, improving the physical design efficiency of very large scale integrated circuits. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 are the wiring actions (a) action 0, (b) action 1 in an embodiment of the present invention;

[0035] Figure 2 are the wiring actions (a) action 2, (b) action 3 in an embodiment of the present invention;

[0036] Figure 3 is an X - structure obstacle - bypassing Steiner tree in an embodiment of the present invention;

[0037] Figure 4 is the overall flowchart of the algorithm in an embodiment of the present invention;

[0038] Figure 5It is the overall process of the DRL router in an embodiment of the present invention;

[0039] Figure 6 It is the algorithmic process of the obstacle bypasser in an embodiment of the present invention;

[0040] Figure 7 It is the first step of obstacle penetration judgment in an embodiment of the present invention;

[0041] Figure 8 It is the second step of obstacle penetration judgment in an embodiment of the present invention;

[0042] Figure 9 It is the third step of obstacle penetration judgment in an embodiment of the present invention;

[0043] Figure 10 It is to judge line segment intersection in an embodiment of the present invention;

[0044] Figure 11 It is the virtual Steiner point selection and obstacle bypass in an embodiment of the present invention;

[0045] Figure 12 It is the pseudo-code of the entire obstacle bypass algorithm in an embodiment of the present invention. Specific implementation manners

[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0047] Please refer to Figures 1-12 , the present invention provides an X-structure obstacle bypassing Steiner minimum tree method based on deep reinforcement learning, including the following three stages:

[0048] Preprocessing stage: The basic wire net information extracted includes the pin coordinate set, the obstacle coordinate set, and the lattice mapping table constructed using the pin coordinates. In order to decompose the multi-pin wire net and simplify the multi-pin wire net problem into a two-pin wire net problem, the algorithm of the present invention uses Prim to construct the MST. As shown in formulas (1) and (2), the Prim algorithm in the present invention not only considers the distance cost between points, that is, uses the sum of the difference in abscissas and the difference in ordinates between two points to represent, but also considers the influence of obstacles on the cost to optimize the obstacle bypass strategy;

[0049] Wiring main stage: Use the MST constructed in the previous stage to establish the environment for the agent training in DRL, and finally use the trained agent to complete the wiring to obtain the XSMT. Then, based on the obstacle bypass strategy, complete the obstacle bypass to obtain the OAXSMT;

[0050] Post-processing stage: Use the DFS algorithm to disassemble the loop of the OAXSMT to obtain the wiring result, and finally calculate the wire length of the wiring result.

[0051] dist = abs(y1 - y2) + abs(x1 - x2) (1)

[0052]

[0053] Among them, the wire network file contains pin and obstacle information

[0054] In this embodiment, the agent adopts a deep Q-network, and the design framework is as follows Figure 5 , obtaining different rewards from the environment to train the agent to find the solution with the highest reward. Specifically, for each two-pin problem, the environment provides state information to the network. Then, the agent estimates the Q-values of each possible action (a0, a1, a2, a3) of the state. Finally, the action to be executed next is selected according to the ε-greedy algorithm and the agent continues to observe the next state in the environment. The method of the present invention calculates the reward according to the reduction in wire length of the currently selected connection action compared to the original connection action and the reduction in wires due to overlap sharing, and at the same time, updates the experience pool. Then, through backpropagation, the data in the experience pool is used to iteratively update the weights of the Q-network. Some key elements of the DQN router will be introduced below.

[0055] State space design: The state is defined as a 4D vector. The first two elements are the rectangular coordinates of the starting point of the agent's current state, and the last two elements are the coordinates of the ending point. Through this encoding strategy, the edge of the MST that the agent is processing can be identified through the current state.

[0056] Action space design: The action represents the wiring method between adjacent nodes in the MST.

[0057] Reward design: The reward is determined according to the current wiring line sharing and wire length reduction obtained by the selected action, as shown in formula (3).

[0058]

[0059] The above design enables the agent to try to select actions with more shared lines and at 45° or 135° when selecting wiring actions. Thus, an XSMT with a shorter wire length is obtained.

[0060] Preferably, in this embodiment, experience replay and target network: By storing the agent's past experiences in the replay buffer and continuously updating the buffer data during training, the network can be better trained. Similarly, by adding a separate target network, the estimation network is updated first each time the network is updated, and the target network is updated only after the estimation network has been updated a certain number of times, which can better learn historical experiences.

[0061] ε-Greedy Strategy: The random strategy randomly selects one of the four connection actions to connect the pins, which can make the agent more daring to try potentially better solutions that have not been discovered. The greedy strategy selects the routing action with the largest Q value based on existing experience. This strategy is defined by the following formula (4):

[0062]

[0063] Network Structure: The Q-network consists of 2 fully connected layers, where the hidden layer contains 32 neurons, followed by a Rectified Linear Unit (ReLU) activation layer. The size of the input layer is 4, which is set to be the same as the length of the state vector. The size of the output layer is 4, which is the same as the size of the action space.

[0064] Preferably, in this embodiment, the obstacle avoidance strategy is specifically:

[0065] By using the XMST learned from the previous stage of DRL learning, first perform an obstacle penetration judgment on the XMST according to the penetration judgment strategy, and then perform a simulated obstacle avoidance to judge the obstacles that the obstacle avoidance route may pass through, so as to obtain all the obstacles that need to be considered when performing obstacle avoidance on the XMST. Then, determine the obstacle avoidance order through iteration until a suitable obstacle avoidance order is found. Then, add virtual Steiner points to the XMST through a certain strategy, and finally complete the obstacle avoidance according to the given four basic connection actions. After that, perform a loop-breaking process on the obstacle avoidance result and calculate the wire length to obtain the final routing result. The details of each part will be described below.

[0066] (1) Penetration Judgment:

[0067] Perform an obstacle penetration judgment on the basic connection actions between points in the XSMT. First, extract the connection actions as individual line segments, and then judge the relationship between the line segments and the obstacles through the following method:

[0068] Judge whether the endpoints of the line segment are inside the rectangular obstacle. If one of the endpoints exists inside, it passes through this obstacle. For example Figure 7 When the situation of (b) occurs, the connection action must penetrate the obstacle. If the two situations of (a) occur, further judgment is required.

[0069] If neither endpoint is inside the obstacle, judge whether the rectangle O1 formed by the line segment intersects with the rectangular obstacle O2. If they do not intersect, the AS does not pass through this obstacle O2. For example Figure 8 When (b) occurs, it means that the line segment AS must not intersect with the obstacle O2. If the two situations of (a) occur, further judgment is required.

[0070] If the above two rectangles intersect, for exampleFigure 9 Judge whether the line segment AS intersects with the main diagonal and the secondary diagonal of the rectangular obstacle, that is, the blue line part. If it intersects, it passes through this obstacle; otherwise, it does not pass through this obstacle. The method of the present invention mainly uses the meaning of cross product to judge whether line segments intersect. For example Figure 9 , take out Figure 9 In (a), the secondary diagonal of the rectangular obstacle O2 is labeled as C3C4. Since the result of the cross product of two-dimensional vectors is a vector parallel to the normal vector of the plane where the two-dimensional vector is located, taking the vector C3C4 as the common vector, if (C3C4 × C3A) · (C3C4 × C3S) < 0, then A and S are on both sides of the line segment C3C4. Similarly, use the same method to judge whether C3 and C4 are on both sides of the line segment AS. If both are satisfied, then C3C4 intersects with AS, which is the case of the right figure in (a) of Figure 8 , and penetration occurs. Otherwise, it is the case of Figure 8 in (b) without penetration

[0071] (1) Simulate obstacle avoidance:

[0072] For the obstacles that have been judged to be penetrated, perform a virtual Steiner point selection for obstacle avoidance in all obstacle avoidance orders. If this simulation of obstacle avoidance is to find an obstacle avoidance route that leads to penetrating a new obstacle, add the penetrated new obstacle to the obstacle avoidance set. If this simulation of obstacle avoidance is to find a suitable obstacle avoidance order, return the current obstacle avoidance order when the simulation of obstacle avoidance is successful

[0073] (2) Selection of virtual obstacle avoidance Steiner points:

[0074] For example Figure 11 , (1) represents the XSMT diagram that needs to complete obstacle avoidance. According to the appropriate obstacle avoidance order, as in (2), first select two points A and B to be connected, calculate the corner point of the four corners of the first obstacle that is the shortest distance to the straight line AB as the first virtual Steiner point, that is, C1, and select the obstacle avoidance action to connect AC1. Next, as in (3), connect C1B, calculate the corner point of the four corners of the second obstacle that is the shortest distance to the straight line C1B as the second virtual Steiner point, that is, C2, select the obstacle avoidance action to connect C1C2, and finally connect C2B to obtain the final result as in (4)

[0075] (3) Ring removal:

[0076] Add the intersection points, virtual Steiner points and pin points of all line segments to the point set, store the connection relationship between each point using the adjacency matrix, traverse the entire graph through DFS, use the visit array to record the access records, and when repeating to access an already visited point, that is, a loop appears, delete this edge, so as to complete the ring removal of the circuit

[0077] The pseudo-code of the entire obstacle avoidance algorithm is asFigure 12 Among them, the first line mainly splits the basic connection action defined in the second chapter between two points. Action 0 is split into two line segments, horizontal and vertical, and action 2 is split into two line segments, an oblique line and a straight edge. At the same time, the position information of the Steiner point of the connection action is analyzed. Lines 2 to 11 complete the obstacle penetration judgment and obstacle avoidance operation. Lines 3 to 7 traverse all obstacles, use the obstacle penetration judgment, and add the obstacles that need to be bypassed to the set for recording. The main function of line 8 is to complete the simulated obstacle avoidance, mainly using the above-mentioned selection of virtual obstacle avoidance Steiner points to perform obstacle avoidance. During this process, potential obstacles that need to be bypassed are judged and added to the obstacle set. The obstacle avoidance sorting in line 9 mainly tries the obstacle set in each edge in turn according to the x coordinate of the lower left point, the y coordinate of the lower left point, the reverse order of the y coordinate of the lower left point, the x coordinate of the upper right point, the y coordinate of the upper right point, and the reverse order of the y coordinate of the upper right point until a reasonable sorting order is found. Line 10 completes the final obstacle avoidance, which is the same as the simulated obstacle avoidance operation. Line 12 analyzes and stores the obtained obstacle avoidance tree in the adjacency matrix, and uses DFS to perform loop breaking processing and cut some unnecessary edges. Line 13 outputs the final OAXSMT line length and draws the relevant results.

[0078] The above are only the preferred embodiments of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of the present invention.

Claims

1. A method for an X - structure obstacle - avoiding Steiner minimum tree based on deep reinforcement learning, characterized in that It includes the following three stages: The preprocessing stage, which simplifies the multi-pin net problem based on the minimum spanning tree algorithm and constructs the MST; The main routing stage, which uses the MST to establish the environment for agent training in the DRL. Finally, the trained agent is used to complete the routing to obtain the XSMT, and then the obstacle avoidance is completed according to the obstacle avoidance strategy to obtain the OAXSMT; The post-processing stage, which uses the DFS algorithm to break the loop of the OAXSMT to obtain the routing result and calculates the wire length of the routing result; The specific obstacle avoidance strategy is as follows: According to the XMST, first perform an obstacle penetration judgment on the XMST according to the penetration judgment strategy, and then perform a simulated obstacle avoidance to judge the obstacles that the obstacle avoidance line may pass through, so as to obtain all the obstacles that need to be considered when performing obstacle avoidance on the XMST. After that, the obstacle avoidance order is determined through iteration until a suitable obstacle avoidance order is found; Then, virtual Steiner points are selected according to the preset requirements and added to the XMST; Finally, the obstacle avoidance is completed according to the given four basic connection actions; For the penetration judgment strategy, perform a penetration judgment on the basic connection actions between points in the XSMT. First, extract the connection actions into individual line segments, and then judge the relationship between the line segments and the obstacles through the following methods: Judge whether the endpoints of the line segment are inside the rectangular obstacle. If one endpoint exists inside, it passes through this obstacle; If neither endpoint is inside the obstacle, judge whether the rectangle O1 formed by the line segment intersects with the rectangular obstacle O2. If they do not intersect, the AS does not pass through the rectangular obstacle O2; If the above two rectangles intersect, judge whether the line segment AS intersects with the main diagonal and the secondary diagonal of the rectangular obstacle, that is, the blue line part. If they intersect, it passes through this obstacle, otherwise it does not pass through this obstacle; The specific simulated obstacle avoidance is as follows: For the obstacles that have been judged to pass through, perform a selection of virtual Steiner points for obstacle avoidance according to all obstacle avoidance orders. If this simulated obstacle avoidance is to find the obstacle avoidance line and causes passing through new obstacles, add the new passed obstacles to the obstacle avoidance set. If this simulated obstacle avoidance is to find a suitable obstacle avoidance order, return the current obstacle avoidance order when the simulated obstacle avoidance is successful.

2. The method for the X-structure obstacle-avoiding Steiner minimum tree based on deep reinforcement learning according to claim 1, wherein, In the preprocessing stage, the basic wire net information extracted includes the pin coordinate set, the obstacle coordinate set, the lattice mapping table constructed using the pin coordinates, and the Prim algorithm is used to construct the MST.

3. The Steiner minimum tree method for obstacle avoidance of X structure based on deep reinforcement learning according to claim 1, characterized in that, The agent adopts a deep Q-network, and by simulating the interaction between the agent and the environment, different rewards are obtained from the environment to train the agent to find the solution with the highest reward. Specifically: For each two-pin problem, the environment provides state information to the network; Then, the agent estimates the Q-values of each possible action (a0, a1, a2, a3) of the state; Finally, the next action to be executed is selected according to the ε-greedy algorithm and continue to observe the next state in the environment.

4. The X-structure obstacle avoidance Steiner minimum tree method based on deep reinforcement learning according to claim 3, characterized in that, The preset elements in the deep Q-network include The state space, where the state is defined as a 4D vector. The first two elements are the rectangular coordinates of the starting point of the agent's current state, and the last two elements are the coordinates of the ending point; The action space, where the action represents the routing method between adjacent nodes in the MST; Reward is determined based on the current routing line sharing and wire length reduction obtained from the selected action, as shown in Equation (3).

5. The method for the X-structure obstacle avoidance Steiner minimum tree based on deep reinforcement learning according to claim 3, wherein The ε-greedy algorithm is specifically as follows: Randomly select a connection action from the four actions to connect the pins, enabling the agent to be more daring in attempting potentially better solutions that have not been discovered. The greedy strategy is to select the routing action with the largest Q value based on existing experience, and this greedy strategy is defined by the following Equation (4):

6. The method for X-structure obstacle avoidance Steiner minimum tree based on deep reinforcement learning according to claim 3, characterized in that The deep Q-network contains 2 fully connected layers. The hidden layer contains 32 neurons, followed by a rectified linear unit activation layer; the size of the input layer is 4, which is set the same as the length of the state vector; the size of the output layer is 4, which is the same as the size of the action space.

7. The method for X-structure obstacle avoidance Steiner minimum tree based on deep reinforcement learning according to claim 1, wherein The loop-breaking process is specifically as follows: Add the intersection points, virtual Steiner points, and pin points of all line segments to the point set, store the connection relationships between points using an adjacency matrix, traverse the entire graph through DFS, and use the visit array to record the access records. When a point that has already been visited is repeatedly accessed, a loop appears, and the edge corresponding to the loop is deleted, thus completing the loop-breaking of the circuit.

Images